Implement GitOps delivery with Argo CD and signed revisions
Install pinned Argo CD 3.4.5, constrain an AppProject, verify repository transport and GnuPG-signed commits or tags, preview manual syncs, sequence waves and hooks, harden revision overrides, rehearse rollback, rotate keys, and plan the separate 3.5 sourceIntegrity migration.
Build a production GitOps path in which Git content, signer identity, repository origin, destination scope, synchronization authority, runtime health, recovery, and audit evidence are independently verifiable.
- Argo CD 3.4.5
- Kubernetes supported by Argo CD 3.4.5
- Git GnuPG commit and annotated-tag signing
- Migration target Argo CD 3.5 sourceIntegrity (planning only)
- Reviewed Kubernetes target and recovery access A supported cluster, explicit Argo CD namespace owner, target namespace owner, Kubernetes RBAC, admission baseline, backup, and independently tested API access exist.
kubectl cluster-info && kubectl auth can-i get applications.argoproj.io -n {{argoCdNamespace}} - Pinned Argo CD artifacts The v3.4.5 installation manifest, CLI, images, digests, provenance, and availability model are reviewed and staged before apply.
sha256sum --check /secure-staging/argocd-v3.4.5/argocd-v3.4.5-reviewed.sha256 - Protected Git release workflow The canonical repository, production branch/tag, required reviews and CI, force-push/tag rules, release signer authorization, and emergency process are documented.
git -C /reviewed/repository remote -v && git -C /reviewed/repository branch -vv - Authenticated public signer fingerprint The full GnuPG fingerprint, expiry, revocation status, owner, and public-only export are confirmed through an authenticated channel independent of the repository.
gpg --with-colons --show-keys {{signerPublicKeyPath}} | awk -F: '$1=="fpr" {print $10}' - Read-only repository credential and server identity A repository-scoped fetch-only credential plus authenticated SSH host key or TLS CA are available through approved secret handling.
test -r {{repositoryCredentialPath}} && test "$(stat -c %a {{repositoryCredentialPath}})" = 600 - Reviewed application manifests The selected Git path renders deterministically and has reviewed namespaces, kinds, images, service accounts, hooks, health, resources, network access, and data compatibility.
- Release and rollback window Operators can freeze or monitor the production reference, disable automation at its authoritative source, preserve evidence, and perform service-level verification.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A pinned Argo CD 3.4.5 control plane whose installation inputs, controller images, repository transport, project scope, permissions, and audit trail can be reviewed independently of the application it deploys.
- A production AppProject that accepts one exact Git repository, one destination cluster and namespace, a deliberately narrow set of Kubernetes resource kinds, and only revisions signed by explicitly approved GnuPG keys.
- A manual-first delivery path that previews generated manifests and diffs, refuses unsigned or incorrectly signed revisions, orders dependencies with sync waves, runs bounded hooks, and exposes health before automation is considered.
- A controlled automation and recovery policy that distinguishes auto-sync, prune, self-heal, revision override, Git revert, and Argo CD history rollback instead of treating them as interchangeable switches.
- An evidence-backed migration plan from the legacy 3.4 project-level signatureKeys mechanism to the sourceIntegrity policies introduced in Argo CD 3.5, without placing 3.5-only fields in a functioning 3.4 manifest.
- The application reaches Synced and Healthy only from the configured repository and a revision whose commit or tag signature matches an imported public key and the AppProject allowlist.
- An unsigned commit, an annotated tag without its own accepted signature, a disallowed repository, an out-of-scope destination, and an unauthorized revision override all fail closed with observable reasons.
- Operators can show the reviewed diff, resolved commit SHA, signer fingerprint, sync phase, wave order, hook result, resource health, deployment history, Kubernetes events, and Argo CD audit evidence for a release.
- A rollback rehearsal demonstrates both preferred Git revert and history rollback, including the mandatory step of disabling automated sync before using argocd app rollback.
- Key rotation overlaps old and new public keys, proves the new signer before removing the old one, and preserves a tested emergency path that does not weaken signature enforcement silently.
Architecture
How the parts fit together
Git remains the desired-state authority, but Git access alone is not sufficient authority to deploy. Argo CD 3.4.5 fetches a pinned repository through an authenticated and host-verified transport, renders one reviewed path, and checks the resolved revision against a project-level list of GnuPG key IDs. The AppProject then constrains where the manifests came from, where they may be applied, and which resource kinds they may contain. Argo CD compares the rendered desired state with the live Kubernetes API and initially waits for an operator to inspect and approve the sync. Sync waves and hooks sequence dependencies, while health checks decide whether the operation may advance. RBAC separates application read, sync, project administration, GPG key administration, and revision override. Automation is introduced only after the negative tests and rollback drill pass. In 3.4, signatureKeys is the working declaration. Argo CD 3.5 adds sourceIntegrity with repository-specific GPG policies and verification modes; that migration is planned here but must occur only with the 3.5 CRDs and controllers.
- Inventory the cluster, current Argo CD ownership, repository, production reference, signing keys, destination, application dependencies, and recovery responsibilities before changing any object.
- Acquire the exact 3.4.5 installation manifest and CLI through the approved artifact process, verify provenance or independently reviewed digests, and apply the complete pinned manifest set.
- Prove component readiness, version consistency, API reachability, transport trust, and a non-admin operator identity before registering production credentials.
- Create a restrictive AppProject and repository connection, then import public verification keys from an authenticated out-of-band source. Never upload a signing private key to Argo CD.
- Enable legacy 3.4 signatureKeys enforcement, create the Application with automated sync explicitly disabled, and run positive and negative signature tests for the selected revision style.
- Render, diff, dry-run, and manually synchronize one signed revision. Observe wave order, hook execution, health, resolved SHA, and post-deployment verification.
- Tighten RBAC and require override privilege whenever a user supplies a revision different from the Application specification.
- Choose automation boundaries explicitly. Prune, self-heal, and allow-empty remain off until their individual failure modes and ownership implications are accepted.
- Rehearse Git revert and Argo CD history rollback with auto-sync disabled, then document key rotation, audits, alerts, and the 3.5 sourceIntegrity migration as separate controlled changes.
Assumptions
- The target is a non-ApplicationSet-managed, single-source Git Application running on a supported Kubernetes cluster. Multi-source Applications, templated AppProject fields, ApplicationSet Git generators, and Helm repositories require separate source-integrity analysis.
- Argo CD 3.4.5 is the reviewed current version for this procedure. Commands and manifests intentionally use 3.4 behavior. The 3.5 sourceIntegrity examples are described only as a migration target and are never applied to the 3.4 control plane.
- The organization controls the Git repository, protected production reference, GPG signing policy, repository deploy credential, and public-key distribution channel. A signed Git object proves possession of a signing key; it does not prove that review, testing, or authorization happened.
- The repository contains declarative Kubernetes manifests, Kustomize, or another built-in renderer that has already been reviewed. Config-management plugins and arbitrary build scripts materially change the execution boundary and are outside this guide.
- Repository credentials, SSO client secrets, webhook secrets, signing private keys, backup encryption keys, and cluster credentials are delivered through approved secret systems. None are committed to Git, pasted into tutorial commands, stored in shell history, or included in screenshots.
- The operator has a tested recovery path to the Kubernetes API that does not depend solely on Argo CD, and can disable automation, inspect the Application, and restore the control plane during an incident.
- Cluster admission and Kubernetes RBAC enforce workload safety after Argo CD authorization. AppProject restrictions are a necessary GitOps boundary but not a complete cluster security policy.
Key concepts
- Desired revision versus resolved revision
- targetRevision can be a moving branch or tag name. Argo CD resolves it to a concrete Git object and records the commit SHA. Reviews and incident records must name the resolved SHA, not only the friendly reference.
- GnuPG signature enforcement in Argo CD 3.4
- A legacy project-wide constraint declared through AppProject.spec.signatureKeys. Argo CD imports public keys into its keyring and refuses revisions that are unsigned, invalid, signed by an unknown key, or signed by a key not listed for the project.
- Commit and tag verification target
- A branch, HEAD, commit SHA, or lightweight tag resolves to a commit, so Argo CD verifies the commit signature. An annotated tag resolves to a tag object, so the annotated tag itself must be signed. A signed commit beneath an unsigned annotated tag does not satisfy that target.
- AppProject boundary
- The intersection of allowed repositories, destination clusters and namespaces, resource kinds, signature keys, and project roles. Wildcards expand authority and must be treated as explicit security decisions.
- Manual sync preview
- A comparison and dry-run that reveals rendered resources, differences, pruning candidates, and validation failures before an operator starts the write operation. It does not replace admission checks that occur only during a real apply.
- Sync phase, wave, and hook
- Hooks assign lifecycle phases such as PreSync and PostSync. The sync-wave annotation orders resources numerically within a phase. Argo CD advances only after the current wave is applied and healthy, so incorrect health or a long-running hook can block later waves.
- Automated sync
- A controller decision to reconcile a new OutOfSync revision without a human sync action. It is separate from prune, selfHeal, and allowEmpty, each of which changes deletion or drift behavior.
- Revision override
- A sync request that supplies a revision different from the Application targetRevision. With application.sync.requireOverridePrivilegeForRevisionSync enabled, this requires the powerful applications override permission in addition to sync.
- Git revert versus history rollback
- A Git revert creates a new signed desired-state commit and preserves the GitOps authority. argocd app rollback re-applies a deployment history entry and cannot run while automated sync is enabled; Git may still point elsewhere afterward.
- sourceIntegrity in Argo CD 3.5
- The successor policy under AppProject.spec.sourceIntegrity. It supports repository-scoped Git GPG policies and verification modes. It requires a 3.5 control plane and CRDs and cannot coexist with signatureKeys in the same project.
Before you copy
Values used in this guide
{{argoCdVersion}}Exact reviewed Argo CD release for the working installation and CLI.
Example: v3.4.5{{argoCdNamespace}}Namespace that owns the Argo CD control plane.
Example: argocd{{projectName}}AppProject name representing one production ownership boundary.
Example: payments-production{{applicationName}}Argo CD Application name used in commands and audit records.
Example: payments-api{{repositoryUrl}}Canonical SSH or HTTPS URL accepted by sourceRepos and registered with Argo CD.
Example: ssh://git@git.example.net/platform/payments-deploy.git{{repositoryPath}}Reviewed path inside the repository containing the production manifests.
Example: environments/production/payments-api{{productionRevision}}Protected Git reference declared by the Application. Use a full reference where ambiguity matters.
Example: refs/heads/production{{targetNamespace}}Only application namespace accepted by the AppProject destination.
Example: payments-prod{{targetServer}}Exact Kubernetes API server URL known to Argo CD for the destination cluster.
Example: https://kubernetes.default.svc{{signerFingerprint}}Full GnuPG fingerprint independently verified for the authorized release signer. Do not rely on a short key ID.
Example: 0123456789ABCDEF0123456789ABCDEF01234567{{signerPublicKeyPath}}Local path to the authenticated ASCII-armored public key used only for import.
Example: /secure-transfer/release-signing-public.asc{{repositoryCredentialPath}}secretEphemeral path to a read-only repository deploy key or credential file. Its contents are secret and must be removed after registration.
Example: /run/user/1000/argocd-repo-readonly{{reviewedHistoryId}}Deployment history ID mapped to an exact reviewed revision and proven compatible with current data before a rollback drill.
Example: 0Security and production boundaries
- Import only public GnuPG keys into Argo CD. A signing private key on the GitOps control plane collapses the separation between authoring and verification and turns a controller compromise into release-signing authority.
- Argo CD treats every imported public key as trusted. It does not implement a Web of Trust or require imported keys to be signed. Verify the full fingerprint through an authenticated channel before import and again before project enforcement.
- A valid signature proves that an accepted key signed the Git object. It does not prove pull-request approval, CI success, vulnerability review, or correct human intent. Keep protected branches, required reviews, status checks, and separation of duties.
- Use exact sourceRepos, destinations, and resource allowlists. A wildcard source plus cluster-wide destination and unrestricted cluster resources gives the project broad deployment authority even if commits are signed.
- Repository credentials should be read-only and repository-scoped. Verify SSH host keys or TLS roots; never solve trust errors with insecure skip-verification in production.
- Treat applications override as near-administrative power for the affected project. It can synchronize a different revision or arbitrary local manifests, so normal release operators should not receive it.
- Do not enable auto-prune, self-heal, or allow-empty merely because automated sync is enabled. Each changes the blast radius of repository mistakes, resource ownership conflicts, or emergency cluster edits.
- Hook containers execute in the target cluster with their assigned service account. Pin images by digest, restrict RBAC and networking, set deadlines, produce bounded logs, and never print credentials.
- Store audit exports outside the Argo CD namespace. A namespace deletion, control-plane compromise, or history retention limit should not erase the only release evidence.
- Redact repository URLs if they embed usernames, internal hostnames when necessary, token-bearing commands, ConfigMap data that may reveal policy structure, and all Secret values before sharing incident evidence.
Stop before continuing if
- Stop if the exact Argo CD release, installation manifest, CLI artifact, or container provenance cannot be independently verified.
- Stop if an existing Argo CD installation, namespace, CRD, AppProject, repository Secret, or Application is owned by another deployment system or team and no handoff has been approved.
- Stop if the full signing-key fingerprint cannot be confirmed through an authenticated channel, if a private key is found in the import material, or if the signer is expired, revoked, or outside the release policy.
- Stop if repository transport requires disabling SSH host-key or TLS certificate verification, or if the only available repository credential has write access broader than the selected repository.
- Stop if the AppProject requires wildcard repositories, destinations, or cluster resources without an approved threat model and admission controls.
- Stop if preview contains unexpected namespaces, cluster-scoped resources, Secrets, webhooks, CRDs, deletion candidates, generated names, mutable images, or a renderer/plugin not covered by review.
- Stop if signature negative tests succeed, if the displayed resolved SHA differs from the reviewed SHA, or if a revision override works for a normal sync-only operator.
- Stop if a hook is privileged, unbounded, dependent on uncontrolled networking, or able to mutate resources outside the target namespace.
- Stop if automated sync cannot be disabled through the actual source of the Application, especially when another controller or ApplicationSet reconciles it.
- Stop rollback after any failed checkpoint; preserve the failed operation, logs, events, Git object IDs, and cluster state before another reconciliation obscures the cause.
instruction
Map the trust boundary and preserve a recovery path
Record who owns the cluster, Argo CD namespace, Git repository, production reference, signing keys, project, destination namespace, workload identities, release approval, and rollback. Capture the current state before installation or adoption. A GitOps controller connects several authority systems; an incomplete ownership map is a deployment risk, not paperwork debt.
Why this step matters
The first control is knowing whether this is a new installation, an adoption of an existing one, or a change to an actively reconciled system. Existing CRDs and a long-lived namespace indicate that uninstalling, reapplying, or taking ownership can affect other applications. A read-only repository remote reduces accidental writes during inspection. Recovery access to Kubernetes must be tested before Argo CD becomes the normal operating path because a broken repository, SSO provider, controller, or project policy can otherwise lock operators out of the very system needed to repair it.
What to understand
Name the accountable owner for the Git reference, signing-key lifecycle, repository credential, AppProject, Argo CD control plane, Kubernetes namespace, application service, and incident command. One person may hold several roles, but the responsibilities must be explicit.
Inventory every existing Application and AppProject in the Argo CD namespace. Record whether Applications are generated, whether another Git repository manages Argo CD itself, and whether a platform controller will overwrite direct edits.
Record the protected branch or tag rules, required reviewers and checks, allowed signing keys, tag immutability, force-push policy, and emergency merge procedure. Argo CD cannot compensate for an undefined repository governance model.
Capture the current cluster context, server URL, namespaces, CRD versions, controller images, and recent namespace events. Store evidence in the approved incident or change record, not only in terminal scrollback.
Confirm an independently authenticated Kubernetes path and namespace backup. A GitOps rollback does not restore deleted CRDs, lost Argo CD Secrets, repository credentials, or an unavailable Kubernetes API.
System changes
- No cluster or repository state changes. This step creates an off-cluster ownership, baseline, and recovery record.
Syntax explained
kubectl config current-context- Prints the selected client context so evidence cannot be mistaken for another cluster.
git remote -v- Shows fetch and push endpoints. A review workspace should not retain an unnecessary production push credential.
status --short --branch- Reports branch, upstream, and uncommitted files without changing the worktree.
Values stay on this page and are never sent or saved.
kubectl config current-context && kubectl cluster-info && kubectl get ns {{argoCdNamespace}} {{targetNamespace}} --show-labels && kubectl get crd applications.argoproj.io appprojects.argoproj.io 2>&1 || true && git -C /reviewed/repository remote -v && git -C /reviewed/repository status --short --branchproduction-eu Kubernetes control plane is running at https://api.production.example.net:6443 NAME STATUS AGE LABELS argocd Active 210d app.kubernetes.io/part-of=argocd payments-prod Active 410d owner=payments NAME CREATED AT applications.argoproj.io 2026-01-12T09:14:28Z appprojects.argoproj.io 2026-01-12T09:14:28Z origin ssh://git@git.example.net/platform/payments-deploy.git (fetch) origin DISABLED-WRITE-URL (push) ## production...origin/production
Checkpoint: Ownership and recovery are complete before mutation
kubectl auth can-i get applications.argoproj.io -n {{argoCdNamespace}} && kubectl auth can-i get deployments -n {{targetNamespace}}Continue whenBoth checks return yes for the approved operator identity, and the change record names every authority owner and recovery route.
Stop whenThe context is unexpected, another system owns the resources, recovery access is untested, repository governance is unknown, or any secret appears in captured output.
If this step fails
Argo CD CRDs exist but the namespace or owner is unclear.
Likely causeA previous installation, cluster bootstrap tool, or managed platform installed shared components.
Inspect CRD labels, annotations, managedFields, and release inventory.Search approved infrastructure repositories for the namespace and CRD names.
ResolutionPause and obtain an ownership handoff. Do not reapply or delete shared CRDs to make the tutorial match the cluster.
Security notes
- Do not capture kubeconfig contents, repository credentials, Secret data, bearer tokens, or private signing keys.
- Use an observer identity for inventory. Installation and project changes should use a time-bounded elevated identity recorded in the change system.
Alternatives
- For a new lab cluster, create the same ownership and rollback notes even if one operator fills every role; the record becomes the production migration checklist.
Stop conditions
- Stop if the Kubernetes context, Git remote, or existing controller ownership differs from the approved change.
command
Stage and verify the pinned Argo CD 3.4.5 release
Acquire the complete Argo CD 3.4.5 installation manifest and matching CLI through the approved release-artifact process. Verify the downloaded files and release provenance before applying anything. The example checksum file represents a value reviewed and stored by the organization; never invent or learn a checksum from the same untrusted download response.
Why this step matters
A version in a URL is necessary but not sufficient supply-chain verification. TLS protects the connection to an endpoint; it does not independently prove that the reviewed release produced the bytes or that a compromised release workflow did not publish them. Argo CD publishes guidance for verifying artifacts and images with provenance. The installation manifest must also be reviewed as a complete set because releases may change RBAC, deployments, CRDs, configuration defaults, and images beyond a simple image tag. Server-side dry-run lets the API server validate the manifests and reveal conflicts before mutation, while preserving the exact staged bytes for later audit.
What to understand
Use an isolated staging directory with restrictive permissions. Retain the manifest hash, provenance result, retrieval time, official release identifier, and reviewer identity in the change record.
Obtain the expected manifest digest or a reviewed artifact bundle through an independently authenticated organizational process. A checksum downloaded beside the file from the same potentially compromised location provides corruption detection but weak independent authenticity.
Verify the matching Argo CD CLI artifact according to the official release-artifact instructions. Record argocd version --client output and its artifact digest before it is permitted in the operator environment.
Review image references, cluster roles, webhooks, CRDs, namespace assumptions, replica counts, Redis mode, ingress exposure, and high-availability requirements. The default non-HA manifest is appropriate only when its availability model is accepted.
Run server-side dry-run with the same namespace and flags intended for the real apply. force-conflicts can transfer field ownership, so inspect conflicts and existing managedFields instead of assuming the flag makes adoption safe.
System changes
- Downloads reviewed public artifacts into a protected local staging directory.
- Performs Kubernetes server-side dry-run only; no cluster object is persisted by this step.
Syntax explained
--proto '=https' --tlsv1.2- Restricts curl to HTTPS and a minimum TLS version for this retrieval.
--fail --location- Fails on HTTP error status and follows reviewed redirects rather than saving an error page as a manifest.
sha256sum --check- Compares staged bytes with the independently reviewed digest file.
--dry-run=server --server-side- Asks the actual API server to validate a server-side apply without persisting it.
--force-conflicts- Matches the official whole-manifest upgrade approach for large CRDs, but can take field ownership and therefore requires review before real use.
Values stay on this page and are never sent or saved.
mkdir -p /secure-staging/argocd-v3.4.5 && cd /secure-staging/argocd-v3.4.5 && curl --proto '=https' --tlsv1.2 --fail --location --output install.yaml https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.5/manifests/install.yaml && sha256sum --check argocd-v3.4.5-reviewed.sha256 && kubectl apply --dry-run=server --server-side --force-conflicts -n {{argoCdNamespace}} -f install.yaml >/tmp/argocd-dry-run.txt && tail -n 8 /tmp/argocd-dry-run.txtinstall.yaml: OK customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io serverside-applied (server dry run) customresourcedefinition.apiextensions.k8s.io/appprojects.argoproj.io serverside-applied (server dry run) serviceaccount/argocd-application-controller serverside-applied (server dry run) deployment.apps/argocd-repo-server serverside-applied (server dry run) deployment.apps/argocd-server serverside-applied (server dry run) statefulset.apps/argocd-application-controller serverside-applied (server dry run)
Checkpoint: Exact artifacts and server-side preview are accepted
cd /secure-staging/argocd-v3.4.5 && sha256sum --check argocd-v3.4.5-reviewed.sha256 && grep -R "image:" install.yaml | sort -uContinue whenChecksum verification succeeds, every image and manifest is associated with v3.4.5 or an explicitly reviewed dependency, and the server dry-run contains no unexplained conflict or validation error.
Stop whenThe digest, provenance, version, image set, server-side validation, ownership, or availability model cannot be verified.
If this step fails
The checksum passes but provenance verification fails.
Likely causeThe expected digest proves consistency with a local record but not the official build origin, or the wrong provenance bundle was used.
Compare artifact name, release tag, source URI, and provenance subject.Retrieve official verification instructions through a separate trusted session.
ResolutionDo not apply the artifact. Resolve the provenance mismatch or obtain an independently approved package.
Security notes
- Do not pipe a remote manifest directly into kubectl in production. Staging creates a reviewable, hashable, and repeatable artifact.
- Remove network proxy credentials and Authorization headers from verbose curl logs.
Alternatives
- Use a reviewed internal artifact mirror and admission policy that verifies Argo CD image provenance. Preserve the upstream release and digest mapping.
- Use the official HA manifest when the accepted availability model requires it; do not mix pieces of HA and non-HA manifests ad hoc.
Stop conditions
- Stop if the manifest references mutable or unexpected images, requests unapproved cluster authority, or conflicts with another owner's fields.
command
Install the complete pinned control plane and verify readiness
Apply the already staged and reviewed 3.4.5 manifest, wait for every controller, and prove that the running images and client/server versions match the accepted release. This is a control-plane change and should run only inside the approved maintenance window.
Why this step matters
Argo CD is highly privileged relative to the resources it manages. A partially ready installation can accept configuration while repository rendering, application reconciliation, or API authorization is unavailable. Waiting on each workload and then inspecting actual images catches incomplete rollouts, admission mutations, image drift, and mixed-version states. The namespace apply is idempotent, but adopting an existing namespace still requires the ownership decision made earlier. A real installation must also configure ingress or port-forward access, TLS, SSO, backups, NetworkPolicy, resource requests, and high availability according to the organization's architecture; those concerns must not be hidden by a successful default rollout.
What to understand
Apply the complete reviewed manifest set. Do not update only container tags because CRDs, arguments, service accounts, RBAC, and configuration may change between releases.
Wait for server, repo-server, application controller, Applicationset controller when installed, Dex when used, Redis, and notification components. Inspect restarts and recent warning events even if rollout status says success.
Compare actual image digests with admission or registry evidence. A tag match alone does not guarantee the runtime image digest remained the reviewed one.
Use a secure local port-forward for initial administration if production ingress and SSO are not ready. Do not expose the API server publicly with a temporary insecure service.
Change or disable the initial administrator credential according to official user-management guidance after SSO and break-glass access work. Never put the bootstrap secret in shell history or the change record.
System changes
- Creates or adopts the Argo CD namespace.
- Creates or updates cluster-scoped CRDs and RBAC plus namespaced Argo CD workloads and services.
- Starts controllers that can reconcile Applications once they are created.
Syntax explained
create namespace --dry-run=client -o yaml | apply- Produces an idempotent namespace declaration while preserving normal apply ownership.
rollout status --timeout=10m- Waits for the declared workload rollout and fails rather than waiting indefinitely.
custom-columns- Prints only runtime identity evidence without dumping environment variables or Secret references.
Values stay on this page and are never sent or saved.
kubectl create namespace {{argoCdNamespace}} --dry-run=client -o yaml | kubectl apply -f - && kubectl apply --server-side --force-conflicts -n {{argoCdNamespace}} -f /secure-staging/argocd-v3.4.5/install.yaml && kubectl rollout status deployment/argocd-server deployment/argocd-repo-server deployment/argocd-applicationset-controller -n {{argoCdNamespace}} --timeout=10m && kubectl rollout status statefulset/argocd-application-controller -n {{argoCdNamespace}} --timeout=10m && kubectl get pods -n {{argoCdNamespace}} -o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[*].ready,IMAGE:.spec.containers[*].imagenamespace/argocd unchanged customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io serverside-applied deployment.apps/argocd-repo-server serverside-applied deployment "argocd-server" successfully rolled out deployment "argocd-repo-server" successfully rolled out statefulset rolling update complete 1 pods at revision argocd-application-controller-7d95f988f7 NAME READY IMAGE argocd-application-controller-0 true quay.io/argoproj/argocd:v3.4.5 argocd-repo-server-74c6f6d8f9-8z2qk true quay.io/argoproj/argocd:v3.4.5 argocd-server-66ddf6c8fc-4p7bn true quay.io/argoproj/argocd:v3.4.5
Checkpoint: Every Argo CD component is ready on 3.4.5
kubectl get pods -n {{argoCdNamespace}} --field-selector=status.phase!=Running && kubectl get events -n {{argoCdNamespace}} --field-selector=type=Warning --sort-by=.lastTimestamp | tail -n 20Continue whenNo required Pod is outside Running/Ready, all reviewed component images resolve to 3.4.5, and there are no unresolved warning events.
Stop whenAny component is mixed-version, crash-looping, unexpectedly exposed, using an unreviewed image, or blocked by admission/RBAC.
If this step fails
The application controller is ready but repo-server is not.
Likely causeImage pull, volume, Redis, configuration, or security-context failure prevents repository operations.
Describe the repo-server Pod and read bounded current/previous logs.Inspect warning events and volume mounts without printing Secret data.
ResolutionRepair the exact manifest, admission, storage, or image issue and reapply the reviewed release. Do not create production Applications while rendering is unavailable.
Security notes
- The Argo CD namespace contains repository and cluster credentials. Limit namespace read access and audit exec, port-forward, Secret, and ConfigMap operations.
- Do not include Secret data in generic support bundles. Collect resource metadata, conditions, events, image IDs, and redacted logs.
Alternatives
- For an established installation, use the official 3.3-to-3.4 upgrade procedure and a separate high-availability rollout plan rather than treating this as a fresh install.
Stop conditions
- Stop before repository registration if the control plane is not fully ready, version-consistent, backed up, and reachable through the approved administrative path.
config
Create an AppProject with explicit source, destination, and resource bounds
Define one production project before the Application exists. The working Argo CD 3.4 manifest includes the legacy signatureKeys list because that is the enforcement mechanism available in this release. The project avoids source and destination wildcards, denies cluster-scoped resources by default, and allows only workload kinds required by the reviewed repository.
Why this step matters
The AppProject is the primary Argo CD tenant boundary. A signed manifest remains dangerous if it can originate from any repository, target any cluster, create privileged cluster-scoped resources, or deploy into another team's namespace. Exact strings also prevent an Application author from switching to a repository they control. The example deliberately denies every cluster-scoped kind and whitelists a small namespaced set. Secret is included only because the example application requires a non-sensitive configuration Secret; in a mature design, secret material should be produced by an approved secret controller and the Git repository should contain only references. The signatureKeys field is intentionally labeled and stored as 3.4-specific configuration.
What to understand
Replace every example with the canonical values from inventory. SSH URLs, scp-like Git URLs, and HTTPS URLs may represent the same repository to humans but compare differently in policy.
List only the namespaces and cluster server URLs the project owns. If multiple environments exist, use separate projects when signer, approver, destination, or resource authority differs materially.
Start with all cluster resources blacklisted. Add a precise clusterResourceWhitelist only when an application genuinely owns a cluster-scoped type and the operational consequences are approved.
Use namespaceResourceWhitelist as an allowlist. Review admission webhooks, Roles, RoleBindings, PVCs, Ingresses, custom resources, and Jobs separately rather than copying a broad wildcard from examples.
signatureKeys activates enforcement for every Application in this project. Ensure the public key is imported before expecting a signed revision to compare successfully, but create the project declaration through the same reviewed Git process.
Orphan warnings help detect resources in the destination namespace that are not owned by the Application. They do not grant ownership and should not trigger automatic deletion.
System changes
- Creates the AppProject and immediately constrains all Applications assigned to it.
- Enables legacy 3.4 GnuPG enforcement for the listed key fingerprint once repository comparison occurs.
Syntax explained
sourceRepos- Allowlist of repository URLs from which project Applications may render.
destinations- Allowed cluster and namespace pairs; both fields matter.
signatureKeys- Argo CD 3.4 project-wide GnuPG key IDs accepted for revision verification.
clusterResourceBlacklist '*/*'- Denies all cluster-scoped kinds unless a reviewed exception is added.
namespaceResourceWhitelist- Restricts namespaced resource kinds the project can deploy.
/reviewed/argocd/payments-project-3.4.yamlapiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: payments-production
namespace: argocd
labels:
owner: payments-platform
spec:
description: Signed production delivery for the payments API
sourceRepos:
- ssh://git@git.example.net/platform/payments-deploy.git
destinations:
- namespace: payments-prod
server: https://kubernetes.default.svc
signatureKeys:
- keyID: 0123456789ABCDEF0123456789ABCDEF01234567
clusterResourceBlacklist:
- group: "*"
kind: "*"
namespaceResourceWhitelist:
- group: ""
kind: ConfigMap
- group: ""
kind: Service
- group: ""
kind: ServiceAccount
- group: ""
kind: Secret
- group: apps
kind: Deployment
- group: batch
kind: Job
- group: networking.k8s.io
kind: NetworkPolicy
orphanedResources:
warn: truekubectl apply --server-side --dry-run=server -f /reviewed/argocd/payments-project-3.4.yaml && kubectl diff -f /reviewed/argocd/payments-project-3.4.yaml || true && kubectl apply -f /reviewed/argocd/payments-project-3.4.yamlappproject.argoproj.io/payments-production serverside-applied (server dry run) diff -u -N /tmp/LIVE /tmp/MERGED + sourceRepos: + - ssh://git@git.example.net/platform/payments-deploy.git + signatureKeys: + - keyID: 0123456789ABCDEF0123456789ABCDEF01234567 appproject.argoproj.io/payments-production created
Checkpoint: Project authority matches the written boundary exactly
kubectl get appproject {{projectName}} -n {{argoCdNamespace}} -o yaml && argocd proj get {{projectName}}Continue whenOnly the approved repository, destination, namespaced kinds, and full signing fingerprint appear; no wildcard expands production authority.
Stop whenAny required wildcard, unexpected cluster resource, external destination, unverified key, or conflicting owner remains.
If this step fails
A required manifest kind is denied by the project.
Likely causeThe repository contains an undeclared dependency or the allowlist is incomplete.
Render and inventory every apiVersion/kind from the reviewed path.Determine whether another platform component should own the denied resource.
ResolutionAdd a narrowly reviewed kind or move the dependency to its proper owner. Do not replace the allowlist with wildcards.
Security notes
- Treat AppProject edit permission as security administration. A user who can broaden sourceRepos or destinations can bypass application-level assumptions.
- Never include Secret data in the project or repository. The presence of Secret in the kind allowlist is not permission to commit credentials.
Alternatives
- Use one AppProject per environment or business boundary to isolate signers, repositories, resource kinds, sync windows, and operator roles.
Stop conditions
- Stop if the repository cannot render within the declared resource kinds or if the project must gain unreviewed cluster-wide authority.
command
Register a read-only repository with verified transport identity
Verify the Git server identity independently, add the approved SSH known-host key or private TLS CA to Argo CD, then register a repository-scoped read-only credential. Keep credential material in an ephemeral protected file, never a command argument, manifest, shell trace, or screenshot.
Why this step matters
A signed commit retrieved from the wrong server is not the intended release. SSH host keys and TLS certificate roots bind repository transport to an authenticated server; repository credentials bind Argo CD to the allowed repository. Production must not use insecure skip-verification because that turns transport errors into a man-in-the-middle opportunity. A deploy key should have fetch-only authority for one repository. Argo CD stores the credential in a Kubernetes Secret, so the namespace access policy and backup system become part of credential protection. The local ephemeral copy must be removed after registration without printing it.
What to understand
Obtain the SSH host fingerprint from the Git service administrator or authenticated service documentation, not by blindly accepting ssh-keyscan output from the same network path.
For HTTPS with a private CA, use argocd cert add-tls with the reviewed CA certificate. Do not use --insecure-skip-server-verification outside a disposable lab.
Use a repository-specific deploy key or narrowly scoped token with read-only contents permission. Do not reuse a developer's personal credential or a token that can push tags or modify branch rules.
The credential file mode should be 0600, owned by the operator, stored on an encrypted or ephemeral filesystem, and excluded from backups and terminal recording.
argocd repo get proves Argo CD can fetch through the registered trust and credential. It does not prove project GPG enforcement; that is tested separately with an Application.
System changes
- Adds an SSH known-host or TLS CA record used by Argo CD repository connections.
- Creates a namespaced Argo CD repository Secret containing the read-only credential reference/data.
- Securely removes the local credential file after successful registration.
Syntax explained
cert add-ssh --batch- Imports reviewed known-host entries without an interactive acceptance ambiguity.
--ssh-private-key-path- Reads the key from a protected file instead of exposing it as an argument value.
--project- Associates the repository credential with the intended AppProject scope.
shred -u- Removes the local temporary file; on copy-on-write or network filesystems secure overwrite is not guaranteed, so prefer ephemeral encrypted storage.
Values stay on this page and are never sent or saved.
argocd cert add-ssh --batch /reviewed/git-server-known-hosts && argocd repo add {{repositoryUrl}} --name payments-production-readonly --ssh-private-key-path {{repositoryCredentialPath}} --project {{projectName}} && argocd repo get {{repositoryUrl}} && shred -u {{repositoryCredentialPath}}Successfully created 1 SSH known host entry Repository 'ssh://git@git.example.net/platform/payments-deploy.git' added TYPE NAME REPO INSECURE OCI LFS CREDS STATUS MESSAGE PROJECT git payments-production-readonly ssh://git@git.example.net/platform/payments-deploy.git false false false true Successful payments-production
Checkpoint: Repository fetch succeeds without weakening transport trust
argocd repo get {{repositoryUrl}} && kubectl get secrets -n {{argoCdNamespace}} -l argocd.argoproj.io/secret-type=repository -o custom-columns=NAME:.metadata.name,PROJECT:.data.projectContinue whenStatus is Successful, insecure verification is false, project scope is correct, credential is read-only, and no local credential file remains.
Stop whenThe host key or CA cannot be authenticated, registration requires insecure mode, credential can write, credential output was exposed, or repository scope is broader than approved.
If this step fails
Repository status is Failed with permission denied.
Likely causeThe deploy key is absent from the repository, has wrong format/permissions, or the URL user/path is incorrect.
Test repository read access in an isolated client without shell tracing.Confirm deploy-key scope and canonical URL with the Git administrator.
ResolutionIssue a new repository-scoped read credential through the approved secret process and repeat registration; do not grant write permission.
Security notes
- Argo CD repository Secrets are sensitive even if read-only. They may reveal internal URLs and permit access to proprietary manifests.
- Disable shell xtrace and command history for credential setup; rotate immediately if a credential appears in logs or screen recordings.
Alternatives
- Use a cloud workload identity or Git provider application supported by the organization's Argo CD design, provided it remains read-only and repository-scoped.
Stop conditions
- Stop if transport identity or read-only authorization cannot be proven without insecure flags.
command
Authenticate and import only the release-signing public key
Inspect the public key in an isolated GnuPG home, compare its full fingerprint and status with an authenticated release-policy record, then import it into Argo CD. The private half must never be present. Argo CD's trust model is intentionally simple: once imported, the public key is trusted for cryptographic verification, so fingerprint authentication is the human control.
Why this step matters
A short key ID is not a safe identity because collisions and lookalikes are practical. The full fingerprint must be compared over an authenticated channel independent of the repository and file transfer used for the key. The isolated GNUPGHOME prevents the operator's personal trust database and keys from influencing inspection. Argo CD accepts ASCII-armored or binary public keys and stores them in argocd-gpg-keys-cm; repo-server reconstructs a transient keyring from that configuration. The displayed TRUST value is not a Web of Trust decision—Argo CD simply trusts imported keys for its configured verification. The key's organizational authorization, expiration, revocation, and custody remain external policy.
What to understand
Obtain the public key and full fingerprint from two controlled channels when possible, such as the security-owned key registry plus a signed release-policy record.
Check key creation, expiry, revoked status, signing capability, UID context, and full fingerprint. A technically valid but expired or deauthorized key should not be imported.
Scan the input with gpg --import-options show-only before argocd gpg add. Confirm it contains no secret key packets and only the intended public identity.
Grant gpgkeys create/delete permissions only to a key-administrator role, separate from routine application sync. Listing can be available more broadly for verification.
After import, allow time for propagation and verify every repo-server replica. Never edit /app/config/gpg/keys directly because the in-pod keyring is transient.
System changes
- Adds the public key to Argo CD configuration and causes repo-server keyrings to synchronize.
- Does not import or handle any signing private key.
Syntax explained
GNUPGHOME=$(mktemp -d)- Creates an isolated temporary key database for inspection.
--import-options show-only --import- Parses and displays key material without adding it to the temporary keyring.
--with-colons- Produces machine-readable GnuPG output so the full fingerprint can be extracted consistently.
argocd gpg add --from- Imports the reviewed public key into Argo CD's managed GPG configuration.
Values stay on this page and are never sent or saved.
GNUPGHOME=$(mktemp -d) && chmod 700 "$GNUPGHOME" && gpg --homedir "$GNUPGHOME" --import-options show-only --import {{signerPublicKeyPath}} && gpg --homedir "$GNUPGHOME" --with-colons --show-keys {{signerPublicKeyPath}} | awk -F: '$1=="fpr" {print $10}' && argocd gpg add --from {{signerPublicKeyPath}} && argocd gpg get {{signerFingerprint}} && rm -rf "$GNUPGHOME"pub rsa4096 2026-06-01 [SC] [expires: 2027-06-01]
0123456789ABCDEF0123456789ABCDEF01234567
uid Payments Release Signing <release-signing@example.net>
0123456789ABCDEF0123456789ABCDEF01234567
GPG key added
KEY ID TYPE SUBTYPE OWNER TRUST CREATED EXPIRES
0123456789ABCDEF0123456789ABCDEF01234567 pub rsa4096 Payments Release Signing unknown 2026-06-01 2027-06-01Checkpoint: The authenticated full fingerprint is present in every verifier
argocd gpg list | grep {{signerFingerprint}} && kubectl get pods -n {{argoCdNamespace}} -l app.kubernetes.io/name=argocd-repo-server -o nameContinue whenExactly the authenticated public fingerprint is listed, no private key was handled, and each ready repo-server has synchronized configuration.
Stop whenFingerprint, expiry, revocation, identity, packet type, propagation, or organizational authorization is uncertain.
If this step fails
argocd gpg add rejects the key or records an unexpected key ID.
Likely causeThe file contains multiple keys, is corrupt, uses an unexpected format, or the operator compared only a subkey/short ID.
Inspect packets and every primary fingerprint in an isolated GNUPGHOME.Ask the signing-key owner for a fresh public-only export.
ResolutionDo not transform unknown material until it imports. Obtain a clean public-only export and re-authenticate the full primary fingerprint.
Security notes
- Public keys are not confidential, but their authorization is security-critical. An attacker who can add a trusted key can make their signatures deployable.
- Never upload a secret-key export, revocation private material, passphrase, or hardware-token backup to Argo CD.
Alternatives
- Manage argocd-gpg-keys-cm declaratively through a separate security-owned GitOps path, provided the public key, fingerprint review, RBAC, and rotation process remain explicit.
Stop conditions
- Stop if the key file contains more than the approved public identity or if the fingerprint cannot be authenticated.
verification
Prove commit, lightweight tag, and annotated tag behavior before deployment
Verify the exact production reference locally and create controlled positive and negative test references in a non-production test repository or protected test branch. A branch, commit SHA, or lightweight tag requires a signed commit. An annotated tag requires a signed tag object. Record object type, resolved SHA, full signer fingerprint, and verification result.
Why this step matters
Teams often say 'the tag is signed' without identifying which Git object Argo CD verifies. That ambiguity causes production failures or false confidence. A branch name, HEAD, commit SHA, and lightweight tag resolve directly to a commit, so the commit's GnuPG signature is checked. An annotated tag is a distinct object and Argo CD checks the tag signature. The peeled revision syntax shows the underlying commit but must not be used to argue that an unsigned annotated tag is acceptable. Positive and negative tests prove actual Argo CD enforcement: an unsigned commit should compare with a ResourceComparison error, an accepted signed commit should render, a lightweight tag should follow its commit, and an unsigned annotated tag above a signed commit should fail.
What to understand
Run local verification in a clean clone with an authenticated signer keyring. Record the exact object type before choosing verify-commit or verify-tag.
For a branch, capture both the moving reference and resolved commit SHA at approval time. A later force-push or fast-forward changes what the same branch name means.
For an annotated release tag, run git verify-tag on the tag object. A signed commit below an unsigned annotated tag does not satisfy Argo CD's target.
For a lightweight tag, Git reports the referenced object as a commit and Argo CD verifies that commit. Lightweight tags do not carry a separate tagger message or signature.
Use a non-production project/repository to create unsigned and wrong-key cases. Do not weaken the production project or push intentionally unsigned test objects to the production reference.
A Good signature only shows cryptographic validity against the local keyring. Compare the full fingerprint to release authorization and check that the Git object contains the reviewed tree.
System changes
- Fetches public Git objects into the review clone and performs read-only cryptographic verification.
- Negative objects, when created for testing, belong only in a dedicated non-production repository/reference.
Syntax explained
cat-file -t- Reports whether the reference resolves directly to a commit or an annotated tag object.
rev-parse ref^{}- Peels a tag to the underlying commit SHA for evidence; it does not verify the tag signature.
verify-commit- Validates the GnuPG signature carried by a commit object.
verify-tag- Validates the GnuPG signature carried by an annotated tag object.
Values stay on this page and are never sent or saved.
git -C /reviewed/repository fetch --tags --prune origin && git -C /reviewed/repository cat-file -t {{productionRevision}} && git -C /reviewed/repository rev-parse {{productionRevision}}^{} && git -C /reviewed/repository verify-commit {{productionRevision}}^{} && git -C /reviewed/repository log -1 --show-signature --format='revision=%H%nsubject=%s%ncommitted=%cI' {{productionRevision}}^{}commit 9b6d5a8f1a90a6dfc2b43b9db27fc4c3496a7d11 gpg: Signature made Mon 27 Jul 2026 14:18:02 UTC gpg: using RSA key 0123456789ABCDEF0123456789ABCDEF01234567 gpg: Good signature from "Payments Release Signing <release-signing@example.net>" [unknown] revision=9b6d5a8f1a90a6dfc2b43b9db27fc4c3496a7d11 subject=Promote payments-api 2026.07.27 committed=2026-07-27T14:18:02+00:00
Checkpoint: Positive and negative Git-object cases behave exactly as documented
git -C /reviewed/repository log -1 --show-signature --format='%H %G? %GF %s' {{productionRevision}}^{}Continue whenThe resolved SHA and full fingerprint match approval; accepted signed objects pass, and unsigned, wrong-key, or unsigned annotated-tag cases are rejected by the test AppProject.
Stop whenObject type, signer, resolved SHA, tree contents, or any negative-test result differs from the written policy.
If this step fails
git verify-commit succeeds locally but Argo CD rejects the branch.
Likely causeThe local keyring trusts a key absent from Argo CD, the project does not list it, the branch moved, or Argo CD fetched another repository URL.
Compare full local fingerprint, argocd gpg list, AppProject signatureKeys, canonical repository URL, and resolved Application SHA.
ResolutionCorrect the proven configuration mismatch or create a new approved release. Do not disable project enforcement.
Security notes
- Do not import private keys into the verification workstation merely to test. Public key material is sufficient.
- Prevent force pushes and tag deletion/movement on production references; signature verification does not make mutable references immutable.
Alternatives
- Target a signed annotated tag for release identity or a protected branch for continuous delivery. Document one policy rather than switching object types casually.
Stop conditions
- Stop if any unsigned or unauthorized object is accepted or any authorized object cannot be tied to the reviewed tree and full fingerprint.
config
Create the Application with automated sync explicitly disabled
Declare the canonical repository, protected reference, reviewed path, project, and destination. Keep automated sync disabled while signature, rendering, project, health, hook, and rollback behavior are tested. Fail on shared resources and create the namespace only when namespace ownership is explicit.
Why this step matters
The Application joins all prior boundaries and starts reconciliation. Explicit false values make the reviewer's intent visible: no commit is applied automatically, missing manifests are not pruned, drift is not self-healed, and an empty render cannot delete everything. CreateNamespace is false because the namespace already exists with separately reviewed labels and policies. FailOnSharedResource stops a sync when another Application already tracks a resource, reducing accidental ownership theft. PruneLast affects only a sync where pruning is actually authorized; it does not enable prune. The finalizer means deleting the Application can delete managed resources, so Application deletion is a destructive operation that requires separate review.
What to understand
Use the same canonical repository URL as the AppProject and repository registration. Small URL differences can cause policy or credential mismatches.
Use a full reference such as refs/heads/production when branch/tag ambiguity matters. Argo CD records the resolved SHA, which must match the signed revision evidence.
Do not put Helm parameters or emergency overrides into the live Application outside Git. They create desired state that is a union of Git and API-side configuration.
Keep destination namespace creation outside the application when namespace labels, quota, NetworkPolicy, Pod security, or ownership are platform responsibilities.
revisionHistoryLimit supports Argo CD history rollback but is not a backup. Preserve Git history and external evidence independently.
An initial OutOfSync/Missing status is expected because nothing has been applied. A signature comparison error is not expected for the approved revision.
System changes
- Creates an Application and starts read-only repository comparison and manifest rendering.
- Does not apply workload resources because automated sync is disabled.
Syntax explained
automated.enabled: false- Prevents controller-initiated sync while retaining explicit fields for future review.
prune: false- Prevents automatic deletion of resources absent from Git.
selfHeal: false- Prevents live-only drift from independently triggering auto-sync.
allowEmpty: false- Preserves protection against a render that produces no resources.
FailOnSharedResource=true- Fails a sync if another Application already tracks a candidate resource.
resources-finalizer.argocd.argoproj.io- Requests managed-resource cleanup when the Application is deleted; deletion is therefore destructive.
/reviewed/argocd/payments-application.yamlapiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: payments-api
namespace: argocd
labels:
owner: payments-platform
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: payments-production
source:
repoURL: ssh://git@git.example.net/platform/payments-deploy.git
targetRevision: refs/heads/production
path: environments/production/payments-api
destination:
server: https://kubernetes.default.svc
namespace: payments-prod
syncPolicy:
automated:
enabled: false
prune: false
selfHeal: false
allowEmpty: false
syncOptions:
- CreateNamespace=false
- FailOnSharedResource=true
- PruneLast=true
- PrunePropagationPolicy=foreground
revisionHistoryLimit: 20Values stay on this page and are never sent or saved.
kubectl apply --server-side --dry-run=server -f /reviewed/argocd/payments-application.yaml && kubectl diff -f /reviewed/argocd/payments-application.yaml || true && kubectl apply -f /reviewed/argocd/payments-application.yaml && argocd app get {{applicationName}} --refreshapplication.argoproj.io/payments-api serverside-applied (server dry run) application.argoproj.io/payments-api created Name: argocd/payments-api Project: payments-production Server: https://kubernetes.default.svc Namespace: payments-prod URL: https://argocd.example.net/applications/payments-api Repo: ssh://git@git.example.net/platform/payments-deploy.git Target: refs/heads/production Path: environments/production/payments-api Sync Policy: <none> Sync Status: OutOfSync from refs/heads/production (9b6d5a8) Health Status: Missing
Checkpoint: Application compares the signed revision without writing
argocd app get {{applicationName}} -o json | jq '{project:.spec.project,source:.spec.source,destination:.spec.destination,automated:.spec.syncPolicy.automated,revision:.status.sync.revision,conditions:.status.conditions}'Continue whenThe project, repository, reference, path, destination, false automation fields, and resolved signed SHA match review; there is no signature or project condition.
Stop whenThe Application auto-syncs, resolves another SHA, contains overrides, reports signature failure for the accepted revision, or targets an unexpected resource.
If this step fails
Application immediately creates resources despite enabled=false.
Likely causeAnother controller manages the Application, the displayed object differs from the authoritative source, or a manual operation was already queued.
Inspect managedFields, owner references, operation state, Application history, and any ApplicationSet.
ResolutionTerminate only a proven unsafe operation, disable automation at the authoritative source, and restore the reviewed Application declaration.
Security notes
- Application create/update permission can redirect a project within its allowed boundary. Restrict it to GitOps/platform automation and audited administrators.
- Deleting an Application with its resources finalizer can delete workloads. Use deletion protection and explicit approval.
Alternatives
- For an app-of-apps or ApplicationSet, model the parent generator and its project templating separately; direct changes to a generated child may not persist.
Stop conditions
- Stop if comparison itself triggers mutation or if the live object does not match its reviewed source.
verification
Render, inventory, diff, and dry-run the signed revision
Refresh the Application, export the rendered manifests, inventory namespaces and kinds, inspect the live diff and pruning candidates, then ask Argo CD for a dry-run. Store a digest of the rendered output with the resolved signed SHA. Do not use --local because GPG-enforced projects intentionally prohibit local-source sync.
Why this step matters
Signature verification answers who signed the Git object; rendering review answers what Argo CD intends to apply. A trusted signer can make an error, a renderer version can change output, a path can include unexpected files, and a Kustomize remote base can expand supply-chain scope. The rendered digest links evidence to exact bytes, although server-side mutation may still change admitted objects. Namespace and kind inventory catches scope escape early. Diff reveals create/update/delete intent, while dry-run exercises Argo CD's operation planning without persisting resources. Local-source sync is intentionally unavailable when signature enforcement is active because local files have no verified Git object.
What to understand
Confirm the Application resolved SHA before and after export. If a moving branch advances during review, discard the preview and repeat against the new signed revision.
Review every rendered image by immutable digest, service account, RBAC object, security context, host access, volume, NetworkPolicy, Service type, ingress, resource limit, probe, and Secret reference.
Search for resources outside targetNamespace and any cluster-scoped kind. The AppProject should reject them, but the unexpected intent still requires repository correction.
Inspect deletion/prune candidates even though automatic prune is false. A later operator can pass --prune manually, so review must classify every extraneous resource.
Keep render artifacts private. ConfigMaps, internal hostnames, annotations, and Secret references may reveal operational data even if Kubernetes Secret values are absent.
Do not assume dry-run runs hooks or external dependencies exactly as a real sync. Admission webhooks, generated values, races, and runtime health remain real-operation gates.
System changes
- Refreshes comparison caches and stores rendered public manifests in a protected evidence path.
- Performs Argo CD diff and operation dry-run only; no workload resource is persisted.
Syntax explained
--hard-refresh- Invalidates manifest and target-state caches so review uses a fresh repository render.
app manifests- Prints the desired manifests generated for the Application's configured source.
--dry-run- Previews the sync operation without affecting the cluster.
tree=detailed- Shows planned resources and hierarchy with operation status.
Values stay on this page and are never sent or saved.
argocd app get {{applicationName}} --hard-refresh && argocd app manifests {{applicationName}} > /secure-staging/{{applicationName}}-rendered.yaml && sha256sum /secure-staging/{{applicationName}}-rendered.yaml && grep -E '^(apiVersion|kind| namespace):' /secure-staging/{{applicationName}}-rendered.yaml && argocd app diff {{applicationName}} --exit-code=false && argocd app sync {{applicationName}} --dry-run --output tree=detailed91bb2a3154598069841c49121563425dbd0ac04c748bb8d2f2d986f847dc0a25 /secure-staging/payments-api-rendered.yaml
apiVersion: v1
kind: ServiceAccount
namespace: payments-prod
apiVersion: apps/v1
kind: Deployment
namespace: payments-prod
apiVersion: v1
kind: Service
namespace: payments-prod
===== apps/Deployment payments-prod/payments-api =====
0a1,8
> spec:
> replicas: 3
GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE
ServiceAccount payments-prod payments-api OutOfSync Missing
apps Deployment payments-prod payments-api OutOfSync Missing
Service payments-prod payments-api OutOfSync Missing
Dry run completed successfullyCheckpoint: Reviewed render, diff, and signed SHA remain identical
argocd app get {{applicationName}} -o json | jq -r '.status.sync.revision' && sha256sum /secure-staging/{{applicationName}}-rendered.yamlContinue whenThe SHA is still the approved signed SHA; rendered digest is recorded; all resources stay within project scope; and no unexplained create, update, delete, hook, or shared-resource conflict remains.
Stop whenThe branch moved, output contains a secret, scope expands, images are mutable, deletion is unexpected, a local source is proposed, or dry-run is not clean.
If this step fails
The rendered manifests change without a Git SHA change.
Likely causeA floating remote base/chart, unpinned tool behavior, plugin side effect, environment-dependent render, or cache inconsistency exists.
Repeat rendering in a clean repo-server and compare tool versions, remote dependencies, and Application parameters.
ResolutionPin every remote dependency and renderer input or replace the nondeterministic build path. Do not deploy an unreproducible render.
Security notes
- Never use argocd app sync --local as a bypass. Enforced GPG projects disallow it because local input has no trusted Git revision.
- Protect rendered evidence like configuration data and securely remove it after the required retention period.
Alternatives
- Run a second independent render in CI and compare its digest with Argo CD output, provided both toolchains and remote dependencies are pinned.
Stop conditions
- Stop for any render nondeterminism, unreviewed scope, mutable dependency, or moved revision.
config
Sequence dependencies with waves and bounded verification hooks
Annotate ordinary resources with numeric sync waves and add only idempotent, least-privilege hooks. The example PreSync Job performs a read-only dependency check before workloads, while the PostSync Job verifies the internal health endpoint after resources become Healthy. Pin hook images by digest and set deadlines, resource limits, and deletion policy.
Why this step matters
Argo CD applies phases in order and waves numerically within a phase, waiting for current-wave resources to become healthy before advancing. That makes health semantics part of deployment control. Negative waves are useful for prerequisites, ordinary resources begin around wave zero, and later waves expose or verify services after workloads are ready. Hooks are cluster workloads, not shell snippets executed by Argo CD. They need their own service accounts, NetworkPolicies, immutable images, deadlines, and logs. A PreSync migration that writes data can make rollback impossible; the example deliberately checks a dependency read-only. PostSync executes only after a successful healthy sync and provides stronger evidence than a green Deployment alone.
What to understand
Use the fewest waves that express real dependencies. Excessive numbering makes the graph hard to review and can hide circular or inaccurate health assumptions.
Ensure Deployment readiness probes represent the service's ability to serve, not only process liveness. PostSync should validate a user-relevant but safe path.
Use stable hook names with BeforeHookCreation so repeated syncs replace old completed hooks predictably. HookSucceeded removes successful Jobs; retain failure evidence until incident capture is complete.
Do not use selective sync for operations that depend on hooks; resource hooks are not run during selective sync.
Keep hook service accounts without token automount unless Kubernetes API access is required. The HTTP checks shown need network access, not API write permission.
A hook that contacts an external service can leak metadata or make deployment availability depend on uncontrolled networks. Allow only reviewed destinations and timeouts.
System changes
- No live workload change in this review step; server-side dry-run validates hook Jobs.
- When the signed Git revision is synchronized, Argo CD will create hook Jobs and apply resources in the declared phase/wave order.
Syntax explained
argocd.argoproj.io/hook- Assigns a resource to lifecycle phases such as PreSync or PostSync.
argocd.argoproj.io/sync-wave- Orders resources numerically within a phase; lower values run first.
BeforeHookCreation,HookSucceeded- Deletes a previous named hook before replacement and removes successful instances while retaining failures for diagnosis.
activeDeadlineSeconds- Bounds total Job runtime so a broken check cannot block reconciliation indefinitely.
environments/production/payments-api/hooks-and-waves.yamlapiVersion: batch/v1
kind: Job
metadata:
name: payments-preflight
namespace: payments-prod
annotations:
argocd.argoproj.io/hook: PreSync
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded
argocd.argoproj.io/sync-wave: "-10"
spec:
backoffLimit: 1
activeDeadlineSeconds: 120
template:
spec:
serviceAccountName: payments-preflight
restartPolicy: Never
containers:
- name: check
image: registry.example.net/operations/http-checker@sha256:REVIEWED_DIGEST
args: ["--url", "https://dependency.example.net/ready", "--timeout", "5s"]
resources:
requests: {cpu: 10m, memory: 16Mi}
limits: {cpu: 100m, memory: 64Mi}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-api
namespace: payments-prod
annotations:
argocd.argoproj.io/sync-wave: "0"
---
apiVersion: v1
kind: Service
metadata:
name: payments-api
namespace: payments-prod
annotations:
argocd.argoproj.io/sync-wave: "10"
---
apiVersion: batch/v1
kind: Job
metadata:
name: payments-smoke
namespace: payments-prod
annotations:
argocd.argoproj.io/hook: PostSync
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded
argocd.argoproj.io/sync-wave: "20"
spec:
backoffLimit: 1
activeDeadlineSeconds: 180
template:
spec:
serviceAccountName: payments-smoke
restartPolicy: Never
containers:
- name: check
image: registry.example.net/operations/http-checker@sha256:REVIEWED_DIGEST
args: ["--url", "http://payments-api.payments-prod.svc.cluster.local/healthz", "--timeout", "5s"]
resources:
requests: {cpu: 10m, memory: 16Mi}
limits: {cpu: 100m, memory: 64Mi}Values stay on this page and are never sent or saved.
git -C /reviewed/repository show {{productionRevision}}:environments/production/payments-api/hooks.yaml | kubectl apply --dry-run=server -f - && argocd app manifests {{applicationName}} | grep -nE 'argocd.argoproj.io/(hook|sync-wave|hook-delete-policy)'job.batch/payments-preflight created (server dry run) job.batch/payments-smoke created (server dry run) 18: argocd.argoproj.io/hook: PreSync 19: argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded 20: argocd.argoproj.io/sync-wave: "-10" 144: argocd.argoproj.io/sync-wave: "0" 231: argocd.argoproj.io/sync-wave: "10" 302: argocd.argoproj.io/hook: PostSync
Checkpoint: Every phase has bounded, observable, least-privilege behavior
argocd app manifests {{applicationName}} | grep -nE 'hook:|sync-wave:|activeDeadlineSeconds:|serviceAccountName:|image:'Continue whenWaves express an acyclic dependency order; hooks are pinned, bounded, idempotent, non-privileged, and have explicit success evidence.
Stop whenA hook writes irreversible state, uses broad RBAC/network access, mutable images, no deadline, secret arguments, or health that cannot distinguish failure.
If this step fails
A later wave never starts even though earlier resources exist.
Likely causeAt least one resource in the current wave is Progressing, Degraded, Unknown, or a hook has not completed successfully.
Inspect argocd app get --show-operation, resource health, Job status, Pods, and events.
ResolutionFix the current wave's health or hook behavior in a new signed revision. Do not skip ahead manually unless the dependency model is proven wrong and change control approves it.
Security notes
- Hook logs must not print tokens, response bodies containing private data, or full request headers.
- Pin hook image digests and admit them through the same provenance policy as application images.
Alternatives
- Use an external progressive-delivery controller for canary analysis, while keeping Argo CD responsible for signed desired state and project boundaries.
Stop conditions
- Stop if any hook or health decision is irreversible, unbounded, overprivileged, or dependent on an untrusted endpoint.
command
Perform the first signed manual sync with an interactive preview
Confirm the Application still resolves the approved signed SHA, start a manual sync with preview, attach the change identifier to the operation, and wait for Synced and Healthy. Do not pass --revision as a convenience; synchronize the revision already declared by the Application.
Why this step matters
The first real sync is the controlled transition from reviewed intent to cluster state. preview-changes requires the operator to inspect and confirm the final diff immediately before mutation. Synchronizing the Application's configured target preserves Git as the declared authority; passing another --revision would create an override path and can deploy a revision that is not represented in spec. The change identifier persists operational context. Waiting for sync, health, and operation separates applied resources from an actually completed and healthy release. Hook results and wave order should match the reviewed plan exactly.
What to understand
Freeze or monitor the production reference during approval. If the resolved SHA changes after preview, answer no and restart verification for the new object.
Review create, update, and delete counts, target namespace, resource kinds, images, hook Jobs, and prune list at the interactive prompt.
Do not add --prune for the first sync unless every deletion candidate has independent approval. The Application's prune=false prevents automatic deletion but a manual flag can still authorize it.
Capture the operation ID, resolved SHA, signer fingerprint, change ID, wave timestamps, hook outcomes, and final resource health.
After Argo CD says Healthy, run service-level verification from an approved client. Health is controller-defined and may not cover external DNS, load balancers, authentication, queues, or business transactions.
If a hook or workload fails, stop and preserve evidence. Repeated blind sync attempts can delete completed hooks, rotate Pods, or obscure the first causal event.
System changes
- Creates or updates the reviewed application resources in payments-prod.
- Runs the bounded PreSync and PostSync Jobs with their declared service accounts and network access.
- Records an Argo CD operation and deployment history entry.
Syntax explained
--preview-changes- Displays changes and waits for explicit operator confirmation before applying.
--info change-id=WYD-302- Persists contextual metadata with the sync operation for audit correlation.
--timeout 900- Bounds CLI waiting time; it does not automatically undo a still-running server operation.
app wait --sync --health --operation- Waits for desired-state convergence, healthy resources, and operation completion.
Values stay on this page and are never sent or saved.
argocd app get {{applicationName}} --refresh && argocd app sync {{applicationName}} --preview-changes --info change-id=WYD-302 --timeout 900 && argocd app wait {{applicationName}} --sync --health --operation --timeout 900 && argocd app get {{applicationName}} --show-operationName: argocd/payments-api Target: refs/heads/production Sync Status: OutOfSync from refs/heads/production (9b6d5a8) Proceed with sync? [y/N] y TIMESTAMP GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE 2026-07-27T15:02:11Z batch Job payments-prod payments-preflight Succeeded PreSync Completed 2026-07-27T15:02:18Z apps Deployment payments-prod payments-api Synced Healthy configured 2026-07-27T15:03:02Z Service payments-prod payments-api Synced Healthy configured 2026-07-27T15:03:06Z batch Job payments-prod payments-smoke Succeeded PostSync Completed Message: successfully synced (all tasks run) Sync Status: Synced Health Status: Healthy Operation Phase: Succeeded
Checkpoint: The approved SHA is Synced, Healthy, and externally verified
argocd app get {{applicationName}} -o json | jq '{revision:.status.sync.revision,sync:.status.sync.status,health:.status.health.status,operation:.status.operationState.phase}' && kubectl get deployment,service -n {{targetNamespace}} -l app.kubernetes.io/instance={{applicationName}}Continue whenResolved SHA equals approval, operation is Succeeded, sync is Synced, health is Healthy, hooks succeeded in order, and service-level verification passes.
Stop whenThe SHA moved, preview differs, any unapproved deletion appears, a hook fails, health degrades, or external verification fails.
If this step fails
CLI times out while the server operation continues.
Likely causeA resource or hook needs longer than the client timeout, health is stuck, or connectivity to the API server was interrupted.
Run argocd app get --show-operation and inspect resource conditions without starting another sync.Check bounded controller and hook logs around the operation timestamp.
ResolutionDiagnose the active operation. Terminate it only when continuation is unsafe and the recovery consequence is understood.
Security notes
- Do not pass auth tokens on the command line. Use the approved CLI context or protected environment injection with short-lived credentials.
- Review operation metadata before sharing; repository URLs and internal resource names may require redaction.
Alternatives
- Use an approved UI sync with the same preview and evidence fields. The control objective is the same: configured revision, explicit approval, bounded operation, and recorded outcome.
Stop conditions
- Stop immediately on revision movement, unexpected deletion, signature condition, hook failure, or health regression.
config
Separate sync from revision override and key administration
Enable application.sync.requireOverridePrivilegeForRevisionSync in argocd-cm, then grant ordinary release operators read and sync for this project but not override, project changes, or GPG key changes. Keep a monitored break-glass role for revision override. Test both identities after configuration reload.
Why this step matters
By default, passing a revision to argocd app sync does not necessarily require the separate override permission. Argo CD 3.2 and later expose application.sync.requireOverridePrivilegeForRevisionSync; setting it true ensures that a sync to a revision other than the Application declaration is treated as override. Override is powerful because it can synchronize another revision or arbitrary local manifests and leave the application OutOfSync. A normal operator needs only get and sync against the configured desired state. GPG key administration must be separate because adding a key changes who can authorize deployments. Project update and Application update permissions are also withheld from these example roles.
What to understand
Merge these keys into existing ConfigMaps rather than replacing unrelated SSO, resource customization, or RBAC configuration. The example file is a focused illustration, not permission to erase current data.
Use SSO groups, not named individuals, for routine roles. Review group membership and ensure the identity provider cannot be self-administered by release operators.
policy.default should be minimal. The example authenticated role has read access only to one project; organizations may choose no default access.
Validate policy syntax before apply and test authorization with dedicated accounts. An admin account can hide missing permissions because it bypasses the intended role.
Test that release-operator can sync the configured revision, cannot sync --revision to another SHA, cannot update the Application or project, and cannot add/remove GPG keys.
Break-glass override needs time-bounded group membership, alerting, ticket correlation, and post-use review. It should not be a standing convenience role.
System changes
- Changes Argo CD server behavior so revision-specified sync requires override privilege.
- Changes Argo CD authorization for application reads, sync, override, and GPG key management.
Syntax explained
application.sync.requireOverridePrivilegeForRevisionSync: true- Treats a supplied alternate revision as an override requiring separate RBAC permission.
applications, sync- Allows synchronization to the Application's declared desired state.
applications, override- Allows powerful alternate manifests or revisions and is withheld from routine operators.
gpgkeys create/delete- Changes the global verifier keyring and belongs to dedicated key administrators.
/reviewed/argocd/argocd-rbac-hardening.yamlapiVersion: v1
kind: ConfigMap
metadata:
name: argocd-cm
namespace: argocd
data:
application.sync.requireOverridePrivilegeForRevisionSync: "true"
---
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-rbac-cm
namespace: argocd
data:
policy.default: role:authenticated
policy.csv: |
p, role:authenticated, applications, get, payments-production/*, allow
p, role:release-operator, applications, get, payments-production/*, allow
p, role:release-operator, applications, sync, payments-production/*, allow
p, role:release-breakglass, applications, get, payments-production/*, allow
p, role:release-breakglass, applications, sync, payments-production/*, allow
p, role:release-breakglass, applications, override, payments-production/*, allow
p, role:key-administrator, gpgkeys, get, *, allow
p, role:key-administrator, gpgkeys, create, *, allow
p, role:key-administrator, gpgkeys, delete, *, allow
g, payments-release-operators, role:release-operator
g, payments-release-breakglass, role:release-breakglass
g, platform-signing-key-admins, role:key-administratorkubectl apply --server-side --dry-run=server -f /reviewed/argocd/argocd-rbac-hardening.yaml && kubectl diff -f /reviewed/argocd/argocd-rbac-hardening.yaml || true && kubectl apply -f /reviewed/argocd/argocd-rbac-hardening.yaml && argocd admin settings rbac validate --policy-file /reviewed/argocd/policy.csvconfigmap/argocd-cm serverside-applied (server dry run) configmap/argocd-rbac-cm serverside-applied (server dry run) configmap/argocd-cm configured configmap/argocd-rbac-cm configured Policy is valid
Checkpoint: Normal operators cannot override revision or trust
argocd account can-i sync applications {{projectName}}/{{applicationName}} && argocd account can-i override applications {{projectName}}/{{applicationName}} && argocd account can-i create gpgkeys '*'Continue whenThe release-operator test identity returns yes for sync and no for override and GPG-key creation; break-glass returns override only during an approved activation.
Stop whenRoutine identities can override revisions, change keys/projects/applications, or gain authority through a broad default or group mapping.
If this step fails
A release operator still syncs --revision successfully.
Likely causeThe configuration key is misspelled, not reloaded, applied to the wrong namespace, or the operator inherits override/admin through another role.
Inspect live argocd-cm, server logs around config reload, effective groups, and all matching RBAC lines.
ResolutionCorrect the config at its authoritative source, remove inherited override, restart only if the documented reload path requires it, and repeat the negative test.
Security notes
- RBAC ConfigMaps reveal group and policy structure; restrict edits and review disclosure.
- Never test denial with a destructive alternate revision. Use a harmless non-production SHA that signature/project policy will also reject.
Alternatives
- Disallow revision override entirely in production and require every emergency revision to be represented by a new signed Git commit updating the Application source.
Stop conditions
- Stop automation rollout until denial tests pass for actual non-admin identities.
decision
Enable only the automation decisions the service can safely recover from
Keep manual sync as the safe default. If change volume and recovery evidence justify automation, enable only automated sync first while leaving prune, selfHeal, and allowEmpty false. Evaluate each additional switch independently; none is an implied best practice for every production service.
Why this step matters
Automation compresses response time but also compresses the time available to detect a bad signed commit. Automated sync applies a new OutOfSync Git revision without a human sync action. Prune allows deletion of objects absent from desired state. selfHeal triggers sync for live drift even without a new Git change. allowEmpty permits an empty desired set to delete all application resources when prune is active. These are separate risk decisions. A signed revision can still be logically wrong, and a compromised authorized signer can produce a valid signature. Therefore rollout begins with manual sync and can progress to automated.enabled=true with destructive and drift behaviors off only after alerts and rollback pass.
What to understand
Make the automation change in the authoritative Git source of the Application. The server-side patch shown is a dry-run preview, not the durable production method.
Automated sync applies each unique commit/parameter combination when OutOfSync under its documented semantics. Failed sync behavior, reconciliation intervals, and retry policy should be monitored.
Keep prune false until every resource has unambiguous ownership, retention, finalizer, stateful-data, and deletion policy. Manual prune still requires explicit approval.
Keep selfHeal false when emergency live intervention is part of the incident plan unless operators can disable automation quickly and coordinate Git. Even with selfHeal false, another source/revision can trigger sync.
Keep allowEmpty false unless deleting the entire application through an empty render is a deliberate tested workflow with independent safeguards.
Automated sync prevents argocd app rollback. The recovery procedure must begin by disabling automation at the source before history rollback can run.
System changes
- The shown command previews an automation change only.
- If committed and synchronized later, enabled=true allows controller-initiated sync of new signed desired revisions while prune, self-heal, and allow-empty remain off.
Syntax explained
enabled- Controls whether Argo CD starts sync automatically for OutOfSync desired state.
prune- Controls automatic deletion of managed resources absent from desired state.
selfHeal- Controls automatic correction of live-only drift.
allowEmpty- Allows automated pruning when the desired resource set is empty; high blast radius.
Values stay on this page and are never sent or saved.
argocd app get {{applicationName}} -o json | jq '.spec.syncPolicy.automated' && kubectl patch application {{applicationName}} -n {{argoCdNamespace}} --type merge -p '{"spec":{"syncPolicy":{"automated":{"enabled":true,"prune":false,"selfHeal":false,"allowEmpty":false}}}}' --dry-run=server -o yaml | grep -A6 automated{
"allowEmpty": false,
"enabled": false,
"prune": false,
"selfHeal": false
}
automated:
allowEmpty: false
enabled: true
prune: false
selfHeal: falseCheckpoint: Automation has a written owner, monitor, and recovery
argocd app get {{applicationName}} -o json | jq '{automated:.spec.syncPolicy.automated,revision:.status.sync.revision,health:.status.health.status}'Continue whenOnly approved fields are enabled, signed-revision and health alerts exist, an operator can disable automation at its source, and rollback has been rehearsed.
Stop whenAutomation is generated elsewhere, disablement is untested, pruning ownership is ambiguous, alerts are absent, or the recovery objective cannot tolerate reconciliation speed.
If this step fails
Automation is re-enabled after an operator disables it.
Likely causeGit, an ApplicationSet, app-of-apps parent, or another controller is the authoritative source and overwrites the live patch.
Inspect managedFields, owner references, Git declaration, and parent controller status.
ResolutionDisable automation in the authoritative source and verify the child Application remains stable before recovery.
Security notes
- Auto-sync is not an authorization mechanism. Preserve branch protection, signature enforcement, project scope, RBAC, and admission controls.
- Alert on changes to Application automation fields and on manual operations performed by break-glass identities.
Alternatives
- Remain manual for production while enabling automation in staging. Promote only signed tags after staging evidence passes.
Stop conditions
- Stop before enabling any automation field whose failure and rollback behavior has not been rehearsed.
command
Rehearse signed Git revert and Argo CD history rollback
Prefer a new signed Git revert because it makes recovery the current desired state. Also rehearse Argo CD history rollback for repository or pipeline emergencies. Before argocd app rollback, automated sync must be disabled at the Application's authoritative source and verified live; Argo CD refuses rollback while auto-sync is enabled.
Why this step matters
Rollback is a production write, not a read-only safety button. The preferred GitOps recovery is a new signed revert commit because branch history then names the recovered desired state and automation can follow it normally. History rollback is useful when Git hosting, CI, or promotion is unavailable, but it applies a previous history entry while the repository may still point to a bad revision. Automated sync would immediately conflict with that intent and Argo CD explicitly disallows rollback while it is enabled. Disabling automation must happen in the source that manages the Application, not only as a live patch that another controller will reverse.
What to understand
Before the drill, choose a harmless reversible application change and define service-level acceptance. Never test rollback by deleting state or weakening security.
For Git revert, verify the bad commit and selected parent, run git revert without rewriting history, sign the new revert commit with an authorized key, review it, and promote through the normal reference.
For history rollback, list history and map the history ID to exact revision, signer, rendered content, schema compatibility, and data migration state. A previous manifest may no longer be safe after irreversible data changes.
Disable auto-sync through Git or generator configuration, wait until the live Application reports enabled=false, then perform rollback. Do not assume selfHeal=false is enough.
Avoid --prune during emergency rollback unless every extraneous resource deletion is required and approved. A previous release may not know resources created by a newer schema or migration.
After history rollback, reconcile Git by creating a signed revert or forward-fix. Leaving live state intentionally different from Git creates an unstable recovery.
Record rollback start/end, reason, history ID, source and target SHA, operator, approval, health, hook behavior, external verification, and restoration of automation.
System changes
- Reapplies manifests from the selected Argo CD deployment history entry.
- May update or recreate namespaced workloads and rerun applicable sync hooks.
- Requires automated sync to remain disabled until Git desired state is reconciled.
Syntax explained
app history- Lists deployment history IDs and revisions; it does not prove runtime/data compatibility.
app rollback APP ID- Synchronizes the Application to a selected previous deployment history entry.
--timeout 900- Bounds client wait time but does not automatically reverse a server operation on timeout.
no --prune- Avoids deleting unexpected resources during the drill unless deletion was separately approved.
Values stay on this page and are never sent or saved.
argocd app get {{applicationName}} -o json | jq '.spec.syncPolicy.automated' && argocd app history {{applicationName}} && argocd app rollback {{applicationName}} {{reviewedHistoryId}} --timeout 900 && argocd app wait {{applicationName}} --sync --health --operation --timeout 900{
"allowEmpty": false,
"enabled": false,
"prune": false,
"selfHeal": false
}
ID DATE REVISION
0 2026-07-27 15:03:06 +0000 UTC 9b6d5a8f1a90a6dfc2b43b9db27fc4c3496a7d11
1 2026-07-27 16:41:22 +0000 UTC 54f21a931d5e4b0d6cc727947043ec36b1d44ef2
TIMESTAMP GROUP KIND NAMESPACE NAME STATUS HEALTH
2026-07-27T17:04:31Z apps Deployment payments-prod payments-api Synced Healthy
Operation Phase: Succeeded
Sync Status: Synced
Health Status: HealthyCheckpoint: Recovery is healthy and Git authority is restored
argocd app get {{applicationName}} --show-operation && argocd app history {{applicationName}} && kubectl rollout status deployment/payments-api -n {{targetNamespace}} --timeout=5mContinue whenThe selected revision is healthy, external verification passes, no unexpected resources were deleted, a signed Git revert/forward-fix represents the recovered state, and automation is restored only after that convergence.
Stop whenAuto-sync is still enabled, history ID is ambiguous, rollback crosses an irreversible schema/data boundary, service health fails, or Git cannot be reconciled.
If this step fails
Rollback succeeds but Application immediately returns OutOfSync.
Likely causeGit still declares the newer revision or another controller changed the Application.
Compare live history revision, spec targetRevision, resolved Git SHA, operation history, and managedFields.
ResolutionKeep automation disabled and land a reviewed signed Git revert or forward-fix representing the recovered state.
Security notes
- History rollback does not re-evaluate the old revision against today's signer authorization, dependency availability, image policy, or security requirements automatically. Re-review it.
- Break-glass access used for rollback must be time-bounded and audited; remove it after recovery.
Alternatives
- Use a signed forward-fix when backward application or data compatibility is uncertain.
- Route traffic away or scale a faulty component through an approved incident control while a signed recovery commit is prepared.
Stop conditions
- Stop rollback if any migration is irreversible, prior artifacts are unavailable/untrusted, or automated reconciliation cannot be held.
instruction
Operate audit evidence and rotate signing keys with overlap
Create recurring evidence and alerts for repository fetch, signature verification, comparison, sync, health, hooks, RBAC, keyring changes, and Kubernetes admission. Rotate signing keys by overlapping old and new public keys, proving a new signed revision, and removing the old key only after every active reference and recovery path is inventoried.
Why this step matters
The signature policy decays if keys expire, staff roles change, references remain signed only by retired keys, or keyring modifications go unobserved. Rotation must avoid two opposite failures: removing the old key too early blocks active releases and recovery, while retaining it indefinitely preserves unnecessary authority. An overlap window lets Argo CD trust both authenticated public keys while a new signed revision proves end-to-end propagation. Inventory every branch, tag, pinned SHA, rollback history, and disaster-recovery artifact before removing the old public key. If a private key is suspected compromised, normal overlap may be unsafe; incident policy decides whether to quarantine, stop sync, revoke, and move immediately.
What to understand
Alert on argocd-gpg-keys-cm changes, AppProject signatureKeys changes, repository credential/certificate changes, Application source changes, automation changes, override use, failed signature comparisons, and admin logins.
Export operation metadata and Kubernetes audit correlation without Secret values. Retain evidence according to security and change-management policy outside the cluster.
Before rotation, authenticate the new full fingerprint and import only its public key. Add the new key to signatureKeys while keeping the old one.
Create a new signed commit or annotated tag using the new key, promote it through the normal protected process, and prove Argo CD accepts it on every repo-server replica.
Inventory active and rollback revisions signed only by the old key. Decide whether to retain the old public key for an approved recovery period or create new signed release objects without rewriting protected history.
Remove the old key from the project first so it no longer authorizes deployments. After monitoring and recovery acceptance, remove it from the global keyring if no project needs it.
Document key expiry alerts at least 90, 60, and 30 days ahead. Emergency rotation under expiry pressure is more error-prone.
System changes
- This evidence command is read-only.
- The documented rotation later adds/removes public keys and updates AppProject signer authorization through reviewed changes.
Syntax explained
gpg list- Inventories public verifier keys currently configured in Argo CD.
proj get- Shows project repositories, destinations, resource policies, and accepted signature keys.
app history- Provides deployment revision history for release and rollback evidence.
Values stay on this page and are never sent or saved.
argocd gpg list && argocd proj get {{projectName}} && argocd app history {{applicationName}} && argocd app get {{applicationName}} --show-operation && kubectl get events -n {{targetNamespace}} --sort-by=.lastTimestamp | tail -n 30KEY ID TYPE OWNER CREATED EXPIRES 0123456789ABCDEF0123456789ABCDEF01234567 pub Payments Release Signing 2026-06-01 2027-06-01 89ABCDEF0123456789ABCDEF0123456789ABCDEF pub Payments Release Signing 2027 2027-04-01 2028-04-01 Name: payments-production Repositories: ssh://git@git.example.net/platform/payments-deploy.git Signature keys: 0123456789ABCDEF0123456789ABCDEF01234567, 89ABCDEF0123456789ABCDEF0123456789ABCDEF ID DATE REVISION 2 2027-04-12 09:22:18 +0000 UTC 6710f55ad4bf318af4d88ec213e7ce97561ad52e Operation Phase: Succeeded
Checkpoint: Audit evidence is complete and new key works before old authority is removed
argocd app get {{applicationName}} -o json | jq -r '.status.sync.revision' && argocd gpg list && argocd proj get {{projectName}}Continue whenThe latest accepted revision is signed by the new authenticated key, both keys propagated during overlap, old-key references are classified, and removal is a separately approved change.
Stop whenNew fingerprint is unverified, propagation is incomplete, an active/rollback reference depends on old authorization unexpectedly, or compromise requires incident escalation.
If this step fails
New-key revision passes locally but fails in only some Argo CD comparisons.
Likely causeRepo-server replicas have inconsistent keyring/configuration or stale caches.
Compare pod revisions, mounted ConfigMap resource versions, keyring listings, and application refresh results across replicas.
ResolutionRepair declarative key configuration and perform a controlled repo-server rollout before removing the old key.
Security notes
- Removing a public key does not revoke leaked private-key signatures in Git history. Repository protection, incident response, revocation publication, and new release objects are separate controls.
- Monitor key-administrator and project-administrator actions independently from routine release operations.
Alternatives
- Use a hardware-backed organizational signing service to reduce private-key extraction risk, provided Git emits a GnuPG signature compatible with the reviewed Argo CD release.
Stop conditions
- Stop normal rotation and invoke incident response if private-key compromise is suspected.
warning
Plan—but do not apply—the Argo CD 3.5 sourceIntegrity migration
Argo CD 3.5 replaces the legacy project signatureKeys declaration with the more capable sourceIntegrity policy. The 3.4 project remains on signatureKeys until a complete 3.5 CRD and control-plane upgrade is validated. Store the future policy only as a migration artifact, never as functioning 3.4 configuration. In 3.5, remove signatureKeys before adding sourceIntegrity because the mechanisms cannot be used together.
Why this step matters
The migration must be explicit because a future-looking manifest can create a dangerous illusion of enforcement on an older controller. In 3.4, signatureKeys verifies the target commit or annotated tag at the project level. In 3.5, sourceIntegrity supports repository-specific Git GPG policies and modes such as head and strict. The upgrade guide says legacy signatureKeys continues temporarily with warnings, but it is deprecated and should be converted. The new and legacy declarations cannot be used together. A safe migration upgrades CRDs and all control-plane components first, proves version consistency, snapshots configuration, exercises one non-production project, converts its declaration, reruns positive and negative tests, and only then schedules production conversion.
What to understand
Keep the 3.5 YAML in a migration document or disabled file clearly marked not applied. Do not merge it into the live 3.4 AppProject.
Read the complete 3.4-to-3.5 upgrade guide for breaking, security, Helm/Kustomize, repository, CRD, and configuration changes—not only the GPG section.
Back up Argo CD resources and configuration and record a downgrade plan. Downgrade requires reintroducing signatureKeys from the sourceIntegrity policies and removing sourceIntegrity.
Apply the complete 3.5 CRDs and controllers in a staging environment, then verify repo-server, application controller, API server, CLI, and all integrations are version-consistent.
Start with mode head to match legacy behavior: verify the target commit/tag. Strict mode verifies the target and ancestors and can fail on older unsigned history, retired keys, or repositories that introduced signing later.
For strict mode, inventory history and key lifecycle. Argo CD 3.5 documents seal commits as a way to establish a verification boundary without rewriting all prior history; evaluate that design separately.
Convert one non-production project by removing signatureKeys and adding sourceIntegrity in one reviewed change. Test signed and unsigned commits, signed and unsigned annotated tags, wrong keys, repository matching, multiple sources if applicable, local-source rejection, and rollback/downgrade.
Only after staging evidence passes, convert production. Monitor deprecated verifyResult consumers and move integrations to structured sourceIntegrityResult where required.
System changes
- No change: the command inspects current versions and CRD schema.
- The future YAML is documentation only and must not be applied to Argo CD 3.4.
Syntax explained
signatureKeys- Working legacy Argo CD 3.4 project-wide key list retained until migration.
sourceIntegrity.git.policies- Argo CD 3.5 policy collection that can scope GPG criteria to repositories.
mode: head- 3.5 mode closest to legacy target-only behavior; shown only in the future migration artifact.
mode: strict- 3.5 mode that verifies target and ancestors; requires history and key-lifecycle preparation and is not silently substituted here.
/reviewed/migrations/argocd-3.5-source-integrity-plan.yaml.not-applied# PLANNING ARTIFACT ONLY.
# Requires Argo CD 3.5 CRDs and controllers. Do not apply to 3.4.
# During the 3.5 migration, remove spec.signatureKeys before adding this field.
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: payments-production
namespace: argocd
spec:
sourceIntegrity:
git:
policies:
- repos:
- url: ssh://git@git.example.net/platform/payments-deploy.git
gpg:
mode: head
keys:
- 0123456789ABCDEF0123456789ABCDEF01234567argocd version --client && argocd version && kubectl get crd appprojects.argoproj.io -o json | jq -r '.spec.versions[] | select(.served==true) | .schema.openAPIV3Schema.properties.spec.properties | keys[]' | grep -E 'signatureKeys|sourceIntegrity' || trueargocd: v3.4.5+reviewed argocd-server: v3.4.5+reviewed signatureKeys
Checkpoint: The live 3.4 project contains only the mechanism it can enforce
kubectl get appproject {{projectName}} -n {{argoCdNamespace}} -o json | jq '{signatureKeys:.spec.signatureKeys,sourceIntegrity:.spec.sourceIntegrity}'Continue whenOn 3.4.5, signatureKeys contains the authenticated keys and sourceIntegrity is null. The 3.5 migration has a test, backup, downgrade, and production change plan.
Stop whensourceIntegrity appears live before 3.5, controllers/CRDs are mixed-version, both declarations coexist, or strict mode is selected without history analysis.
If this step fails
The 3.5 migration accepts unsigned history unexpectedly or rejects all old releases.
Likely causeThe selected policy mode, repository matcher, key list, annotated-tag semantics, or historical trust boundary differs from the intended policy.
Test head and strict separately in staging with known signed, unsigned, wrong-key, and tag fixtures.Inspect structured sourceIntegrityResult and exact resolved Git objects.
ResolutionKeep production on the proven 3.4 legacy policy or 3.5 head compatibility behavior while correcting the policy and history plan. Never disable verification as a migration shortcut.
Security notes
- A field accepted by kubectl is not proof that every running controller enforces it. Version and behavioral negative tests are required.
- Do not publish internal repository URLs, signer fingerprints tied to private organizational identities, or downgrade backups outside authorized systems.
Alternatives
- Remain temporarily on 3.5 legacy compatibility with signatureKeys and deprecation warnings while preparing conversion, within the project's supported-upgrade policy.
Stop conditions
- Stop production migration if CRDs/controllers are inconsistent, downgrade is untested, policy fixtures fail, or any unsigned case is accepted.
Finish line
Verification checklist
argocd version && kubectl get pods -n {{argoCdNamespace}} -o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[*].ready,IMAGE:.spec.containers[*].imageClient and server report the reviewed 3.4.5 release and every required component is Ready on the approved image digest.argocd proj get {{projectName}}Only the canonical repository, exact destination, reviewed resource kinds, and authenticated signature key fingerprints are allowed.argocd repo get {{repositoryUrl}}Connection is Successful with insecure verification false and a read-only project-scoped credential.argocd gpg list && argocd gpg get {{signerFingerprint}}The authenticated public fingerprint is present, current, and propagated; no private signing material exists in Argo CD.git -C /reviewed/repository cat-file -t {{productionRevision}} && git -C /reviewed/repository log -1 --show-signature --format='%H %G? %GF %s' {{productionRevision}}^{}The object type is understood, resolved SHA matches approval, and the full accepted fingerprint verifies the commit or signed annotated tag as appropriate.argocd app get {{applicationName}} -o json | jq '.status.conditions'Dedicated tests prove unsigned, wrong-key, and unsigned annotated-tag revisions fail with a comparison condition and cannot synchronize.argocd app manifests {{applicationName}} | grep -E '^(apiVersion|kind| namespace):'Every manifest stays within the reviewed destination, kind, image, service-account, and hook boundaries.argocd app wait {{applicationName}} --sync --health --operation --timeout 900The approved signed SHA is Synced and Healthy, operation succeeded, waves and hooks completed in order, and external service verification passed.argocd account can-i override applications {{projectName}}/{{applicationName}}The routine release operator receives no; only monitored break-glass authority can override.argocd app get {{applicationName}} -o json | jq '.spec.syncPolicy.automated'Automation fields exactly match the approved mode; prune, selfHeal, and allowEmpty are not enabled implicitly.argocd app history {{applicationName}}History IDs map to reviewed revisions, auto-sync disablement is tested, Git revert is the preferred path, and data/schema compatibility is documented.kubectl get appproject {{projectName}} -n {{argoCdNamespace}} -o json | jq '{signatureKeys:.spec.signatureKeys,sourceIntegrity:.spec.sourceIntegrity}'On Argo CD 3.4.5, signatureKeys is populated and sourceIntegrity is absent; the 3.5 policy remains an unapplied migration artifact.Recovery guidance
Common problems and safe checks
Argo CD reports ResourceComparisonError and says the revision is not signed or cannot be verified.
Likely causeThe resolved commit or tag is unsigned, signed with a different key, malformed, or not accepted by the AppProject signatureKeys list.
Record argocd app get {{applicationName}} -o yaml without changing the Application.Resolve and verify the exact object locally with git rev-parse and git verify-commit or git verify-tag.Compare the full signer fingerprint with argocd gpg list and the AppProject signatureKeys.
ResolutionCreate a new correctly signed commit or signed annotated tag through the reviewed release process, or correct a proven key-distribution mistake. Do not disable enforcement to make the sync pass.
A signed commit passes on a branch but the annotated release tag fails.
Likely causeArgo CD verifies the annotated tag object when targetRevision resolves to an annotated tag; the commit signature below it is not a substitute for a signed tag.
Run git cat-file -t {{productionRevision}} to distinguish tag from commit.Run git verify-tag {{productionRevision}} and inspect the full fingerprint.
ResolutionCreate a new signed annotated tag with git tag -s under the accepted signer policy. Never move or rewrite the existing production tag silently.
argocd gpg list shows the key, but repo-server still rejects a valid signature.
Likely causeThe keyring has not propagated to every repo-server pod, the AppProject lists another key ID, or the running pods are not the expected version/configuration.
Inspect argocd-gpg-keys-cm metadata without exposing unrelated data.List repo-server pods and compare rollout age and image digest.Inspect /app/config/gpg/keys read-only inside each repo-server pod only under approved debug access.
ResolutionWait for documented propagation, repair the declarative ConfigMap source, or perform a controlled repo-server rollout. Never modify the transient in-pod keyring.
The repository connection fails with SSH host-key or TLS certificate errors.
Likely causeThe server identity is absent, changed, or signed by a private CA not configured in Argo CD; a proxy may also be intercepting the connection.
Verify the repository endpoint and certificate or SSH fingerprint through an authenticated administrator channel.Inspect Argo CD repository certificate configuration and repo-server logs with credentials redacted.
ResolutionAdd the reviewed SSH known host or custom TLS CA through Argo CD certificate management. Do not use insecure skip-verification in production.
The AppProject rejects the Application source or destination.
Likely causerepositoryUrl string normalization, destination server URL, namespace, or resource scope does not exactly match the restrictive project rule.
Compare argocd app get output with kubectl get appproject {{projectName}} -n {{argoCdNamespace}} -o yaml.Confirm the canonical repository URL and cluster server identity.
ResolutionCorrect the Application if it is wrong. Expand the project only through a reviewed boundary change; do not add a wildcard as a convenience fix.
Dry-run succeeds, but real sync is denied.
Likely causeAdmission, Kubernetes RBAC, quota, policy, or a mutating webhook differs from client-side rendering and comparison.
Read the failed operation details and Kubernetes events for the exact resource.Use kubectl auth can-i --as the Argo CD controller service account for the denied verb and resource.
ResolutionFix the intended workload or narrowly adjust an independently reviewed admission/RBAC policy. Do not grant cluster-admin to the controller.
Sync is stuck in Progressing before later waves.
Likely causeA resource in the current wave is not Healthy, a hook is still running, or a custom resource lacks an accurate health assessment.
Use argocd app get {{applicationName}} --show-operation and inspect resource health.List hook Jobs, Pods, events, and deadlines in {{targetNamespace}}.
ResolutionCorrect the unhealthy resource or bounded hook and commit a signed fix. Change custom health logic only with tests that cover degraded and unknown states.
A PreSync or PostSync Job repeats unexpectedly.
Likely causeThe hook has a generated name, no appropriate deletion policy, a failed prior instance, or repeated sync attempts at the same revision.
Inspect hook annotations, Job owner references, status, creation times, and Argo CD operation history.Verify idempotency and activeDeadlineSeconds in the Git manifest.
ResolutionMake the hook idempotent, use a stable name or reviewed generateName strategy, and apply an explicit hook-delete-policy. Never rely on manual Job deletion as normal control flow.
A normal operator can sync an arbitrary revision.
Likely causeapplication.sync.requireOverridePrivilegeForRevisionSync is false or missing, or RBAC grants applications override too broadly.
Inspect the exact argocd-cm key and argocd-rbac-cm policy.Test with a dedicated sync-only account against a harmless unsigned or disallowed revision.
ResolutionSet application.sync.requireOverridePrivilegeForRevisionSync to true and remove override from ordinary roles. Re-test after the server configuration reload.
The application reverts emergency live changes even though selfHeal is shown as false.
Likely causeAnother Git revision triggered automated sync, another controller owns the resource, or the Application is multi-source and another source changed.
Inspect Application operation history, managedFields, Git revision changes, and all reconcilers for the resource.Confirm automated.enabled and the actual managing source if the Application is generated.
ResolutionDisable automation at its authoritative source, coordinate the emergency change in Git, and resolve competing ownership. Do not assume selfHeal=false is a universal reconciliation pause.
argocd app rollback is refused.
Likely causeAutomated sync is still enabled, the selected history ID is absent, or RBAC does not authorize the operation.
Read spec.syncPolicy.automated from the live Application and its managing Git/ApplicationSet source.Run argocd app history {{applicationName}} and confirm the exact history ID and revision.
ResolutionDisable auto-sync through the source of truth, wait until it is effective, select the reviewed history entry, and retry. Prefer a signed Git revert when time and repository availability allow.
Prune proposes deleting a shared or stateful resource.
Likely causeResource ownership is ambiguous, a manifest was removed unintentionally, or prune boundaries were enabled before inventory and retention policy were complete.
Inspect Argo CD tracking labels/annotations, managedFields, owner references, PVC policy, and other Applications.Preview the exact prune list without approving the sync.
ResolutionKeep prune disabled, restore or transfer the manifest through review, and define ownership or Delete/Prune confirmation policy before another attempt.
After key rotation, old releases or branches no longer verify.
Likely causeThe old key was removed before all active target revisions were promoted or the new key was not present in every repo-server keyring.
Inventory every Application targetRevision and verify its resolved object signer.Confirm both old and new public keys in Argo CD and both key IDs in the project during overlap.
ResolutionRestore the authenticated old public key temporarily if policy permits, re-sign through a new release commit/tag without rewriting history, and repeat the overlap plan. Never import an unverified replacement key under incident pressure.
Applying a sourceIntegrity field to the 3.4 AppProject fails schema validation or is ignored.
Likely causesourceIntegrity belongs to the Argo CD 3.5 CRDs and controller behavior, not the working 3.4 configuration.
Run argocd version and inspect installed ApplicationProject CRD schemas.Compare the manifest with the release-3.4 GPG documentation and the 3.4-to-3.5 upgrade guide.
ResolutionRemove the future-only field from the 3.4 change, retain signatureKeys, and schedule a complete 3.5 CRD/controller migration before converting the policy.
Reference
Frequently asked questions
Does a green Git-provider Verified badge guarantee Argo CD will accept the revision?
No. Argo CD performs its own GnuPG verification against public keys in its keyring and key IDs allowed by the AppProject. Provider badges may include other signature types or provider-owned keys and are not the Argo CD enforcement result.
Can Argo CD verify SSH or Sigstore Git signatures in 3.4?
The project-level mechanism covered here is GnuPG verification for Git repositories. Do not assume another Git signature format is accepted unless the exact Argo CD release documentation says so and the behavior is tested.
Should production target a branch, commit SHA, lightweight tag, or annotated tag?
A protected branch supports continuous promotion, a commit SHA is immutable but requires updating the Application, a lightweight tag verifies the commit, and an annotated tag verifies the tag object. Select one documented release identity and test its signing semantics end to end.
Why not enable auto-sync, prune, and self-heal immediately?
They automate different decisions. Auto-sync applies a new desired revision, prune deletes resources absent from Git, and self-heal reverses live drift. Each needs ownership, failure, and recovery tests before production.
Can the 3.5 sourceIntegrity YAML be stored now while running 3.4?
It may be kept as a reviewed migration artifact outside the live 3.4 AppProject, but it must not be represented as enforced by 3.4. Apply it only after the 3.5 CRDs and controllers are installed and validated.
Recovery
Rollback
Prefer a new reviewed and GnuPG-signed Git revert so the protected reference represents recovery. If Git or promotion is unavailable, first disable automated sync at the Application's authoritative source, verify enabled=false on the live object, review an Argo CD history ID and its data compatibility, then run argocd app rollback without prune unless deletions are separately approved. Reconcile Git immediately after a history rollback. Control-plane rollback is a different change: restore backed-up Argo CD configuration and use the official version-to-version procedure; never downgrade CRDs/controllers ad hoc.
- Declare an incident/change freeze, capture the current Application, project, keyring, repository status, resolved SHA, operation, resource health, events, and service symptoms without exposing secrets.
- Disable automated sync in the Git/ApplicationSet/app-of-apps source that owns the Application. Wait until the live Application reports automated.enabled=false and prove another controller does not restore it.
- Choose between a signed Git revert, signed forward-fix, traffic mitigation, or Argo CD history rollback based on repository availability, data/schema compatibility, and recovery objective.
- For Git recovery, verify the bad commit and parent, create a non-history-rewriting revert, sign it with an authorized key, review it, promote it, preview the Argo CD diff, and synchronize.
- For history recovery, run argocd app history, map the reviewed ID to exact manifests and signer evidence, then run argocd app rollback {{applicationName}} {{reviewedHistoryId}} --timeout 900 without --prune by default.
- Wait for operation, sync, and health; inspect hooks and waves; verify the service externally and confirm no stateful resource or schema crossed an irreversible boundary.
- If history rollback was used, create a signed Git revert or forward-fix representing the recovered state before re-enabling automated sync.
- Restore automation only after Git, Application status, service health, monitoring, and incident owner all agree. Time-bound and remove break-glass access.
- For key compromise, stop normal rotation, disable unsafe sync paths, preserve evidence, remove the compromised key from project authorization under incident control, import an authenticated replacement public key, and issue a new signed release object.
- For a failed 3.5 migration, follow the tested downgrade plan: restore signatureKeys from policy keys and remove sourceIntegrity before returning to a controller version that lacks it.
Evidence