Monitor Kubernetes with Prometheus, Grafana, and kube-state-metrics
Deploy a pinned kube-prometheus stack, validate target discovery and kube-state-metrics semantics, control cardinality, create actionable rules, route alerts without literal secrets, test notification and resolution end to end, provision dashboards, and operate storage and upgrades.
Provide trustworthy, secure, capacity-bounded Kubernetes monitoring that detects user impact and platform risk, routes actionable alerts to accountable responders, supports evidence-driven diagnosis, and remains testable during failure and upgrade.
- Kubernetes 1.25+, current supported
- kube-prometheus 0.18.0
- Prometheus Operator 0.92.0, 0.92.x
- Prometheus 3.12.0, 3.12.x
- Alertmanager 0.33.0, 0.33.x
- kube-state-metrics 2.19.0, 2.19.x
- Grafana 13.0.2, 13.0.x
- Supported cluster Confirm Kubernetes 1.25 or newer, release compatibility, architectures, admission, network, storage, and provider limitations.
kubectl version - Pinned supply chain Acquire kube-prometheus v0.18.0 and component images through the approved verification and immutable-digest process.
- SLO and paging policy Define user-impact indicators, severity response, owners, runbooks, grouping, inhibition, silence, and test expectations.
- Storage and capacity Provide a reviewed encrypted expandable StorageClass and calculate active-series, samples, WAL, retention-time, retention-size, and query headroom.
- Identity and secret delivery Provide authenticated Grafana and API access plus external delivery for receiver and administrator credentials; no literal credential enters Git or CLI history.
- Canary namespace Use a disposable application and test receiver for discovery, rule, notification, and resolution exercises.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A version-pinned kube-prometheus 0.18 monitoring stack using its integrated Prometheus Operator 0.92.0, Prometheus 3.12.0, Alertmanager 0.33.0, kube-state-metrics 2.19.0, node-exporter, and Grafana 13.0.2 with a documented compatibility and upgrade boundary.
- A target-discovery and metrics model that distinguishes Kubernetes object state from application behavior, limits uncontrolled labels, records expensive service-level queries, and exposes scrape, rule, notification, storage, and dashboard failures.
- An actionable alerting system whose pages identify user impact, urgency, owner, runbook, evidence, and safe first checks; every critical route is tested end to end and dashboards are treated as investigation views rather than paging logic.
- Operators can prove that API-server, kubelet, node, container, application, and kube-state-metrics targets are discovered by the intended ServiceMonitor or PodMonitor selectors and that unexpected targets do not enter the collection boundary.
- A controlled canary creates a real Prometheus alert, passes it through Alertmanager grouping and routing, reaches the test receiver, resolves cleanly, and leaves evidence of rule evaluation, notification latency, labels, and inhibition behavior.
- Dashboards answer capacity, saturation, errors, latency, rollout, availability, and Kubernetes desired-versus-observed-state questions without using unbounded high-cardinality labels or hiding missing-data conditions.
Architecture
How the parts fit together
kube-prometheus combines Prometheus Operator custom resources, Prometheus, Alertmanager, node-exporter, kube-state-metrics, Grafana dashboards, and curated rules. The Operator reconciles Prometheus, Alertmanager, ServiceMonitor, PodMonitor, Probe, PrometheusRule, and AlertmanagerConfig objects; it does not scrape data itself. Prometheus discovers selected monitor objects, scrapes targets, stores time series, evaluates recording and alerting rules, and sends active alerts to Alertmanager. kube-state-metrics converts Kubernetes API object state into metrics without measuring application traffic or node resource usage. Alertmanager groups, deduplicates, inhibits, silences, and delivers notifications. Grafana queries data sources and renders dashboards. Each layer can be healthy while another is broken, so installation, target discovery, rule evaluation, routing, delivery, and user-facing behavior have separate checkpoints.
- Define service-level objectives, paging policy, data retention, expected target count, cardinality and resource budgets, security boundaries, and the recovery or remote-storage decision.
- Pin the kube-prometheus release branch and inspect the generated manifests, CRDs, images, RBAC, host access, network requirements, and version compatibility before applying them.
- Create CRDs and wait for discovery, then apply the remaining stack and verify every component, image, replica, volume, and PodDisruptionBudget.
- Validate monitor selectors and target health, add one controlled application ServiceMonitor, and prove that a mismatched label remains undiscovered.
- Understand kube-state-metrics semantics and add only approved label or annotation allowlists to avoid uncontrolled cardinality and sensitive metadata exposure.
- Create recording and alerting rules with `for`, clear severity, ownership, runbook links, missing-data behavior, and tests tied to actual user or capacity impact.
- Configure Alertmanager routing and inhibition with secret delivery outside source-controlled manifests, then exercise firing, notification, grouping, resolution, and receiver failure.
- Provision Grafana data sources and dashboards as code, protect administrative access, validate query freshness and time range, and operate storage, upgrades, and rollback.
Assumptions
- The Kubernetes cluster is at least 1.25 because current Prometheus Operator CRDs use CEL validation, and the pinned kube-prometheus release supports the actual Kubernetes minor, node operating systems, architectures, and API server.
- A monitoring namespace, persistent storage class, ingress or port-forward access method, DNS and TLS boundary, and recovery path exist before production data retention is accepted.
- Cluster administrators can review CRDs, ClusterRoles, hostNetwork or hostPID use, node-exporter mounts, kubelet access, admission effects, and NetworkPolicies.
- The organization has a paging policy that distinguishes immediate human action from ticket, dashboard, and capacity-review signals. Severity names have documented response expectations.
- Receiver credentials, API tokens, SMTP passwords, and Grafana administrator credentials are delivered by a secret-management mechanism and are never placed in this guide, Helm values, Git, CLI flags, or screenshots.
- Metrics can reveal workload names, namespaces, nodes, tenants, software versions, routes, and business volume. Access and retention follow the same data-classification process as operational logs.
- Application owners define user-impact indicators. Kubernetes desired-versus-observed state is useful but cannot replace latency, errors, correctness, throughput, freshness, or queue-age signals.
- The examples use upstream kube-prometheus release artifacts rather than a third-party hosted monitoring service. Managed alternatives need their own identity, egress, retention, residency, and cost review.
Key concepts
- Prometheus Operator
- A Kubernetes controller that translates monitoring custom resources into managed Prometheus and Alertmanager configuration and workloads.
- kube-prometheus
- An upstream collection of manifests, dashboards, and rules integrating Prometheus Operator, Prometheus, Alertmanager, node-exporter, kube-state-metrics, and Grafana.
- kube-state-metrics
- A service that watches Kubernetes API objects and exposes their desired and observed state as metrics; it is not a node or application performance agent.
- ServiceMonitor
- An Operator custom resource selecting Services and named ports to create scrape targets within the selectors permitted by a Prometheus resource.
- Recording rule
- A periodically evaluated PromQL expression stored as a new time series to make repeated queries cheaper and semantics more consistent.
- Alerting rule
- A PromQL condition that moves from inactive to pending to firing and attaches labels and annotations sent to Alertmanager.
- Alertmanager inhibition
- A rule suppressing a lower-level alert when a related higher-level alert is firing, for example suppressing Pod symptoms during a confirmed cluster outage.
- Cardinality
- The number of distinct label combinations in a metric set. Unbounded identifiers, URLs, user IDs, and arbitrary Kubernetes labels can exhaust memory and storage.
- Missing data
- Absence of samples that may mean success, target failure, discovery failure, rule error, or query mismatch. Every important rule defines which interpretation is correct.
- Actionable alert
- A notification tied to real impact or imminent risk, with owner, urgency, context, runbook, safe first checks, and a condition that naturally resolves.
Before you copy
Values used in this guide
{{kubePrometheusVersion}}Pinned upstream kube-prometheus release.
Example: v0.18.0{{monitoringNamespace}}Namespace containing the monitoring stack.
Example: monitoring{{applicationNamespace}}Namespace containing the monitored canary application.
Example: observability-canary{{applicationService}}Service selected by the application ServiceMonitor.
Example: checkout-metrics{{serviceMonitorLabel}}Label required by the selected Prometheus monitor selector.
Example: monitoring=platform{{retentionTime}}Local Prometheus retention duration after capacity review.
Example: 15d{{retentionSize}}Local Prometheus size cap below persistent-volume capacity.
Example: 80GB{{alertReceiver}}Approved non-production receiver used for route verification.
Example: platform-alert-canary{{runbookBaseUrl}}HTTPS base URL for stable alert runbooks.
Example: https://oneliners.guru/guides{{expectedTargetCount}}Reviewed count or bounded range of active scrape targets after installation.
Example: 42-55{{grafanaHostname}}Authenticated internal Grafana endpoint.
Example: grafana.ops.example.invalid{{receiverCredential}}secretReference to a credential supplied outside configuration; never a literal value.
Example: managed by the platform secret workflowSecurity and production boundaries
- Prometheus endpoints and Grafana must not be exposed anonymously. Protect them with authenticated ingress, TLS, network policy, and least-privilege roles; port-forwarding is suitable only for controlled administration.
- Prometheus Operator and monitoring service accounts often have broad list/watch access. Review namespace selectors and RBAC, and do not let arbitrary tenants create monitor or rule objects consumed by a privileged global Prometheus.
- Metrics and labels can leak tenant names, image versions, nodes, endpoints, repository paths, and business volume. Do not export Kubernetes annotations or labels broadly from kube-state-metrics.
- Receiver and Grafana credentials must come from an external secret workflow or separately managed Kubernetes Secret. Never put literal credentials in AlertmanagerConfig, values files, `--set`, shell history, ConfigMaps, dashboard JSON, or guide variables.
- A Kubernetes Secret is not inherently encrypted at rest and anyone able to create a Pod in its namespace may be able to consume it indirectly. Enable encryption at rest and least-privilege RBAC.
- Rule and dashboard expressions can be computationally expensive. Restrict who can create global rules, review joins and regexes, and measure query duration before production rollout.
- node-exporter requires host visibility. Inspect mounts, host namespaces, capabilities, and node scheduling; do not add privileged access merely to make one collector work.
- Silences can hide real incidents. Require an owner, narrow matchers, bounded duration, reason, and audit; do not use a broad silence as an alert-quality fix.
Stop before continuing if
- Stop if the pinned kube-prometheus release does not support the cluster's Kubernetes minor or required architecture.
- Stop if CRDs, ClusterRoles, host access, images, or remote endpoints cannot be reviewed and pinned before apply.
- Stop if Grafana, Prometheus, Alertmanager, or metrics endpoints would be publicly or anonymously reachable.
- Stop if receiver or administrator credentials would be committed, printed, placed in CLI history, or embedded in generated manifests.
- Stop if target discovery crosses tenant boundaries or permits arbitrary namespaces to inject scrape endpoints and rules into a privileged Prometheus.
- Stop if label cardinality, retention capacity, WAL growth, or persistent-volume recovery has not been load-tested for production scale.
- Stop if a page has no defined human action, owner, runbook, urgency, or resolution condition.
- Stop if the canary alert cannot be fired, delivered, resolved, and audited end to end.
instruction
Define SLO, paging, retention, cardinality, and access budgets
Inventory required signals and targets, define who may query them, calculate storage and active-series budgets, and separate page-worthy symptoms from tickets and dashboards.
Why this step matters
Monitoring design starts with decisions, not manifests. Without SLO and response semantics, a large default rule set can create noisy pages while critical user-impact signals remain absent.
What to understand
List clusters, nodes, workloads, API endpoints, kubelets, storage, network, and application targets with expected counts and owners.
Classify each signal as page, ticket, dashboard, capacity planning, audit, or temporary diagnosis. Define severity response time and escalation.
Estimate samples from target count, series per target, and scrape interval; add headroom for rollouts, label changes, WAL, compaction, and queries.
Define local retention time and size, persistent-volume class, backup or rebuild expectations, and whether long-term remote storage is required.
Document who can create ServiceMonitor, PrometheusRule, AlertmanagerConfig, dashboards, silences, and receivers. Separate tenant contribution from global consumption.
System changes
- No changes; creates a monitoring design and budget.
Syntax explained
kubectl top- Shows current resource use only when Metrics API is available; it is not a sizing history.
kubectl get node -o wide && kubectl get namespace && kubectl get pvc -A && kubectl top nodeNAME STATUS ROLES AGE VERSION worker-1 Ready <none> 120d v1.36.1 NAME STATUS AGE default Active 420d monitoring Active 35d
Checkpoint: Every page and data budget has an owner
Continue whenTarget inventory, SLOs, paging classes, retention, cardinality, capacity, access, and recovery decisions are approved.
Stop whenA critical service lacks user-impact signals or unlimited labels and retention are accepted without capacity evidence.
If this step fails
Teams request every warning as a page.
Likely causeSeverity and human-action policy is undefined.
kubectl get prometheusrule -A
ResolutionRequire each page to state impact, immediate action, owner, runbook, and natural resolution; route the rest to tickets or dashboards.
Security notes
- Do not inventory Secret values or private endpoint tokens; names and access boundaries are enough.
Alternatives
- Begin with one representative cluster and publish measured budgets before scaling globally.
Stop conditions
- Do not install without a retention and access owner.
instruction
Pin and review the upstream kube-prometheus release
Download the signed upstream v0.18.0 release through the approved supply-chain process, record its commit, and review manifests, CRDs, images, privileges, selectors, storage, and compatibility before applying.
Why this step matters
A release branch is a coherent tested set. Copying current main or mixing independent latest images bypasses its compatibility assumptions and makes rollback ambiguous.
What to understand
Use v0.18.0 release artifacts and verify provenance according to organizational policy. Record exact commit and component image digests.
Read the kube-prometheus compatibility matrix and Prometheus Operator requirement for Kubernetes 1.25 or newer.
Review every ClusterRole, CRD, ServiceAccount, host mount, host namespace, Service, NetworkPolicy, storage field, and image registry.
Customize through the project's documented Jsonnet or supported overlay workflow rather than editing generated output with undocumented drift.
Preserve the unmodified release for diff and rollback. Record every deliberate overlay with owner and test.
System changes
- No cluster changes; prepares a pinned reviewed release.
Syntax explained
v0.18.0- Pinned kube-prometheus integration release, not an instruction to float on main.
git -C kube-prometheus describe --tags --always && kubectl version && rg 'image:|hostNetwork:|hostPID:|cluster-admin|persistentVolumeClaim' kube-prometheus/manifestsv0.18.0 Client Version: v1.36.1 Kustomize Version: v5.7.1 Server Version: v1.36.1 manifests/prometheus-prometheus.yaml: image: quay.io/prometheus/prometheus:v3.12.0
Checkpoint: Release content and compatibility are known
Continue whenThe exact release, commit, images, Kubernetes compatibility, privileges, overlays, and rollback artifacts are recorded.
Stop whenAny image floats, unsupported Kubernetes version appears, or generated privileges cannot be explained.
If this step fails
The release references an image blocked by policy.
Likely causeRegistry policy, architecture, signature, or mirror process differs from upstream.
rg 'image:' kube-prometheus/manifests
ResolutionMirror the exact verified digest through the approved process and record the deterministic overlay; do not substitute an unrelated version.
Security notes
- Review source and signatures outside the cluster before applying cluster-scoped resources.
Alternatives
- Use the official Prometheus Operator installation plus separately managed components when the full kube-prometheus opinionated stack is unsuitable.
Stop conditions
- Do not apply an unpinned main-branch snapshot.
command
Install CRDs first, then the pinned monitoring stack
Apply setup resources, wait until the monitoring APIs are discoverable, apply the remaining release, and wait for component readiness without exposing any user interface publicly.
Why this step matters
CRDs must be established before custom resources are accepted. Server-side apply records field ownership, but it does not remove the need to inspect conflicts and release guidance.
What to understand
Apply only the reviewed release directory. Wait for CRDs rather than repeatedly applying until errors disappear.
Inspect server-side conflicts. Do not force ownership over fields controlled by another deployment system without a migration plan.
Wait for Pods, then inspect StatefulSets, Deployments, DaemonSets, Services, EndpointSlices, PVCs, PDBs, and events.
Keep Grafana, Prometheus, and Alertmanager ClusterIP-only or behind authenticated internal access. Use a temporary port-forward for initial administration.
Compare running image IDs with approved digests and verify replicas across failure domains where the design expects HA.
System changes
- Creates monitoring CRDs, namespace, cluster RBAC, Operator, Prometheus, Alertmanager, Grafana, kube-state-metrics, node-exporter, rules, dashboards, Services, and related resources.
Syntax explained
--server-side- Uses API server field management; conflicts require review rather than automatic force.
Established- Confirms the API server serves a CRD before dependent resources are applied.
Values stay on this page and are never sent or saved.
kubectl apply --server-side -f kube-prometheus/manifests/setup && kubectl wait --for=condition=Established crd/prometheuses.monitoring.coreos.com crd/servicemonitors.monitoring.coreos.com crd/prometheusrules.monitoring.coreos.com --timeout=180s && kubectl apply --server-side -f kube-prometheus/manifests && kubectl -n {{monitoringNamespace}} wait --for=condition=Ready pod --all --timeout=600scustomresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com serverside-applied customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com condition met namespace/monitoring serverside-applied pod/alertmanager-main-0 condition met pod/prometheus-k8s-0 condition met pod/grafana-6d9b8f5fd7-k9z2p condition met
Checkpoint: All pinned components are ready and private
Continue whenExpected replicas, image digests, PVCs, CRDs, targets, and Services are healthy with no public unauthenticated endpoint.
Stop whenAny CrashLoop, Pending PVC, field conflict, unexpected LoadBalancer or NodePort, or image drift appears.
If this step fails
Custom resources fail with no matches for kind.
Likely causeCRDs are not established or the release uses a different API version.
kubectl get crd | grep monitoring.coreos.comkubectl api-resources | grep monitoring.coreos.com
ResolutionWait for and validate the pinned setup CRDs; do not downgrade manifests independently.
Security notes
- Cluster-scoped monitoring RBAC is privileged. Keep manifest-apply permission limited and audited.
Alternatives
- Use GitOps server-side apply with the same ordered CRD synchronization and readiness gates.
Stop conditions
- Stop if applying the release would overwrite an existing monitoring owner without planned migration.
verification
Verify component versions, replicas, storage, and internal access
Inspect workloads and image IDs, query build information through controlled port-forwards, verify persistent volumes, and confirm no anonymous external service exists.
Why this step matters
Pod readiness can coexist with wrong versions, ephemeral storage, single-zone placement, public Services, or missing disruption budgets. A component inventory makes those assumptions visible.
What to understand
Compare runtime image IDs to reviewed digests and component versions to the release manifest.
Verify Prometheus and Alertmanager replicas, anti-affinity, PDB behavior, and failure-domain distribution against availability goals.
Check Prometheus PVC class, size, retention, expansion, encryption, and recovery procedure. Grafana persistence is less important when dashboards and data sources are provisioned as code.
List Service types and ingress objects. There should be no accidental public LoadBalancer, NodePort, or anonymous route.
Inspect restarts, OOMKills, throttling, failed mounts, and warning events before target troubleshooting.
System changes
- No changes; validates component implementation.
Syntax explained
imageID- Reports the immutable runtime image identity rather than only a mutable tag.
Values stay on this page and are never sent or saved.
kubectl -n {{monitoringNamespace}} get deploy,statefulset,daemonset,pod,svc,pvc,pdb -o wide && kubectl -n {{monitoringNamespace}} get pod -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .status.containerStatuses[*]}{.imageID}{" "}{end}{"\n"}{end}'NAME READY AGE deployment.apps/prometheus-operator 1/1 8m deployment.apps/kube-state-metrics 1/1 8m deployment.apps/grafana 1/1 8m statefulset.apps/prometheus-k8s 2/2 8m statefulset.apps/alertmanager-main 3/3 8m persistentvolumeclaim/prometheus-k8s-db Bound 100Gi
Checkpoint: Runtime matches the approved architecture
Continue whenVersions, digests, replicas, placement, PVC, PDB, resource state, and internal access match the design.
Stop whenA component runs an unapproved digest, critical storage is ephemeral, or an interface is publicly accessible.
If this step fails
Prometheus Pods are Ready but one has no persistent volume.
Likely causeThe release overlay omitted storage or a volume mount points to emptyDir.
kubectl -n {{monitoringNamespace}} get prometheus k8s -o yamlkubectl -n {{monitoringNamespace}} get pod prometheus-k8s-0 -o yaml
ResolutionApply the reviewed storage specification and migration procedure before accepting retention or HA claims.
Security notes
- Do not include Grafana admin Secret values or receiver configurations in inventory output.
Alternatives
- Use authenticated internal ingress for daily access and reserve port-forward for emergency administration.
Stop conditions
- Do not proceed when runtime differs from release evidence.
config
Add a controlled application ServiceMonitor
Expose one named metrics port through a Service, select it with a narrowly labelled ServiceMonitor, and prove positive and negative discovery boundaries.
Why this step matters
Explicit Services, named ports, monitor labels, and namespace scope make discovery explainable. Broad selectors allow unintended tenants or endpoints to inject data and increase access or cardinality.
What to understand
The Service selects only the application Pods and exposes only the named metrics port. Confirm EndpointSlices contain ready expected addresses.
The ServiceMonitor selector applies to Service labels, not Pod labels. Its own metadata labels must also match the Prometheus resource's serviceMonitorSelector.
Use namespaceSelector matchNames rather than any namespace unless cross-namespace collection is explicitly governed.
Start with a 30-second interval and 10-second timeout. Faster scraping multiplies load and storage; change it from measured requirements.
Create a second monitor with a deliberately wrong label and prove it is not discovered, then remove the negative fixture.
System changes
- Creates a metrics Service and ServiceMonitor, causing Prometheus to add a scrape target when selectors and RBAC match.
Syntax explained
port: metrics- References the Service port name, not the container number directly.
namespaceSelector.matchNames- Restricts discovery to an explicit namespace list.
application-monitor.yamlValues stay on this page and are never sent or saved.
apiVersion: v1
kind: Service
metadata:
name: {{applicationService}}
namespace: {{applicationNamespace}}
labels:
app: checkout
monitoring: platform
spec:
selector:
app: checkout
ports:
- name: metrics
port: 8080
targetPort: metrics
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: checkout
namespace: {{applicationNamespace}}
labels:
monitoring: platform
spec:
selector:
matchLabels:
app: checkout
monitoring: platform
namespaceSelector:
matchNames: [{{applicationNamespace}}]
endpoints:
- port: metrics
path: /metrics
interval: 30s
scrapeTimeout: 10sValues stay on this page and are never sent or saved.
kubectl apply --server-side -f application-monitor.yaml && kubectl -n {{applicationNamespace}} get svc,servicemonitor,endpointslice --show-labelsservice/checkout-metrics serverside-applied servicemonitor.monitoring.coreos.com/checkout serverside-applied NAME TYPE PORT(S) service/checkout-metrics ClusterIP 8080/TCP NAME ENDPOINTS endpointslice.discovery.k8s.io/checkout-metrics 10.42.2.17:8080
Checkpoint: Only the intended target is Up
Continue whenThe exact application endpoints appear as healthy targets with expected labels; a mismatched monitor remains absent.
Stop whenNo target appears, an unrelated endpoint is scraped, or sensitive labels are copied into series.
If this step fails
ServiceMonitor selects no Service.
Likely causeThe Service lacks one metadata label or the selector was mistakenly based on Pod labels.
kubectl -n {{applicationNamespace}} get svc --show-labelskubectl -n {{applicationNamespace}} get servicemonitor checkout -o yaml
ResolutionAlign stable Service metadata labels and keep Pod selection separate.
Security notes
- Do not expose a metrics endpoint carrying secrets, request bodies, raw URLs, user IDs, or tokens.
Alternatives
- Use PodMonitor only when no stable Service abstraction is appropriate and the additional Pod-level discovery is reviewed.
Stop conditions
- Stop if tenant namespaces can create monitors consumed globally without policy.
instruction
Interpret and govern kube-state-metrics
Verify kube-state-metrics version and target health, distinguish object state from resource usage, and permit only reviewed label and annotation dimensions.
Why this step matters
kube-state-metrics reports Kubernetes object fields. Misreading its metrics as application health creates pages that are both noisy and blind to real failures.
What to understand
Use deployment, StatefulSet, DaemonSet, Job, PVC, node condition, resource request, and limit metrics for desired-versus-observed control-plane questions.
Use kubelet and node-exporter for CPU, memory, filesystem, and container usage. Use application metrics and probes for user-visible behavior.
Review metric stability and deprecations during upgrades; record dashboards and rules against the pinned kube-state-metrics version.
Do not enable broad `--metric-labels-allowlist` or annotation allowlists. Arbitrary labels can expose sensitive metadata and create unbounded cardinality.
Measure series count before and after any allowlist change and preserve only stable dimensions required by a named rule or dashboard.
System changes
- No direct changes; establishes semantic and cardinality governance.
Syntax explained
DESIRED/AVAILABLE- Kubernetes controller state that supports rollout diagnosis but does not measure transaction success.
Values stay on this page and are never sent or saved.
kubectl -n {{monitoringNamespace}} get deploy kube-state-metrics -o wide && kubectl -n {{monitoringNamespace}} get servicemonitor kube-state-metrics -o yaml && kubectl get deploy -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,DESIRED:.spec.replicas,AVAILABLE:.status.availableReplicas'NAME READY UP-TO-DATE AVAILABLE kube-state-metrics 1/1 1 1 NAMESPACE NAME DESIRED AVAILABLE observability-canary checkout 3 3
Checkpoint: Object-state semantics and label budget are explicit
Continue whenRules document why kube-state-metrics is appropriate, and no uncontrolled Kubernetes label or annotation becomes a metric label.
Stop whenA user-impact page depends only on desired replicas or an unbounded label allowlist is proposed.
If this step fails
Series count jumps after enabling labels.
Likely causeA deployment, chart, controller, or user creates many unique Kubernetes label values.
kubectl -n {{monitoringNamespace}} get deploy kube-state-metrics -o yamlkubectl get pod -A --show-labels
ResolutionRemove the uncontrolled allowlist dimension, restart through the managed release, and redesign the query around stable labels.
Security notes
- Kubernetes annotations often contain external IDs, repository links, or operator metadata; do not export them by default.
Alternatives
- Join approved ownership metadata from a controlled static mapping rather than exporting arbitrary object labels.
Stop conditions
- Stop any allowlist change without a measured series budget.
config
Create recording rules and one actionable alert
Record a stable deployment availability ratio and alert only when a sustained replica deficit affects the canary, with owner, severity, summary, impact, and runbook annotations.
Why this step matters
A recording rule centralizes semantics and reduces repeated query cost. A sustained alert with clear ownership and impact is easier to operate and test than dozens of raw component symptoms.
What to understand
Use server-side dry run for schema and admission, then rule-unit tests in CI for PromQL semantics. Dry run cannot prove real data behavior.
Aggregate only by stable namespace and deployment labels. Avoid Pod UID, container ID, image digest, URL, request ID, and user dimensions in paging identity.
Use `clamp_min` to avoid division by zero, but separately decide what a scaled-to-zero workload means. Missing series need explicit tests.
`for: 10m` filters brief rollout transitions. Derive duration from service tolerance and deployment strategy, not arbitrary noise suppression.
Annotations explain summary, impact, and runbook; labels drive routing. Keep mutable prose out of routing matchers.
System changes
- Creates a recording rule and alerting rule loaded dynamically by selected Prometheus instances.
Syntax explained
record- Stores the expression result under a stable metric name.
for: 10m- Requires a continuously true condition before firing.
owner- Stable routing and accountability label, not a free-form user name.
platform-rules.yamlValues stay on this page and are never sent or saved.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-workloads
namespace: {{monitoringNamespace}}
labels:
prometheus: k8s
role: alert-rules
spec:
groups:
- name: platform.workloads
interval: 30s
rules:
- record: namespace_workload:deployment_available_ratio
expr: sum by (namespace, deployment) (kube_deployment_status_replicas_available) / clamp_min(sum by (namespace, deployment) (kube_deployment_spec_replicas), 1)
- alert: DeploymentAvailabilityBelowTarget
expr: namespace_workload:deployment_available_ratio{namespace="{{applicationNamespace}}",deployment="checkout"} < 1
for: 10m
labels:
severity: page
owner: platform
annotations:
summary: Checkout has unavailable replicas
impact: Sustained replica loss may reduce capacity or availability.
runbook_url: {{runbookBaseUrl}}/diagnose-kubernetes-pod-failuresValues stay on this page and are never sent or saved.
kubectl apply --server-side --dry-run=server -f platform-rules.yaml && kubectl apply --server-side -f platform-rules.yaml && kubectl -n {{monitoringNamespace}} get prometheusrule platform-workloadsprometheusrule.monitoring.coreos.com/platform-workloads serverside-applied (server dry run) prometheusrule.monitoring.coreos.com/platform-workloads serverside-applied NAME AGE platform-workloads 4s
Checkpoint: Rules load and return expected fixture results
Continue whenOperator accepts the rule, Prometheus reports it healthy, fixture tests cover normal, deficit, scaled-zero, missing, and recovery cases.
Stop whenThe rule is rejected, expensive, high-cardinality, has no owner, or fires on normal rollout fixtures.
If this step fails
Rule is present but not loaded.
Likely causePrometheus ruleSelector labels do not match or Operator rejected the resource.
kubectl -n {{monitoringNamespace}} get prometheus k8s -o yamlkubectl -n {{monitoringNamespace}} describe prometheusrule platform-workloads
ResolutionAlign the narrow release labels and correct validation errors; never broaden selection to all untrusted namespaces.
Security notes
- Runbook URLs should be authenticated where incident context is sensitive; do not put secret values in annotations.
Alternatives
- Use multi-window SLO burn-rate alerts for mature user-facing services rather than raw replica ratios.
Stop conditions
- Do not page on an expression without fixture tests.
config
Configure routing, inhibition, and secret boundaries
Create a narrow AlertmanagerConfig route whose receiver credential is referenced from a separately managed Secret, configure only justified inhibition, and keep resolved notifications enabled for the canary.
Why this step matters
Alertmanager routing is a production control plane. Secret references, stable grouping, and an isolated canary route allow validation without exposing receiver credentials or flooding real responders.
What to understand
The Secret is created separately by the approved secret workflow. The manifest contains only name and key references; do not use `stringData` with a literal URL in Git.
Group by stable incident dimensions. Grouping by Pod or volatile labels creates duplicate notifications during rollouts.
Use groupWait to collect related alerts, groupInterval to update a group, and repeatInterval to remind responders. Tune from incident workflow evidence.
Inhibition should suppress strictly derivative symptoms, not hide independent failures. Test equality labels and recovery ordering.
Receiver templates must escape untrusted labels and avoid copying sensitive annotations to broad channels.
System changes
- Creates an AlertmanagerConfig route and receiver reference; the external secret workflow creates the credential separately.
Syntax explained
urlSecret- References a Kubernetes Secret key without placing the receiver URL in the manifest.
sendResolved- Sends recovery notification so end-to-end resolution can be tested.
groupBy- Defines stable dimensions that become one notification group.
alert-routing.yamlValues stay on this page and are never sent or saved.
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
name: platform-routing
namespace: {{monitoringNamespace}}
labels:
alertmanagerConfig: main
spec:
route:
receiver: {{alertReceiver}}
groupBy: [alertname, cluster, namespace]
groupWait: 30s
groupInterval: 5m
repeatInterval: 4h
matchers:
- name: owner
value: platform
matchType: =
receivers:
- name: {{alertReceiver}}
webhookConfigs:
- urlSecret:
name: platform-alert-receiver
key: url
sendResolved: trueValues stay on this page and are never sent or saved.
kubectl apply --server-side --dry-run=server -f alert-routing.yaml && kubectl apply --server-side -f alert-routing.yaml && kubectl -n {{monitoringNamespace}} get alertmanagerconfig platform-routingalertmanagerconfig.monitoring.coreos.com/platform-routing serverside-applied (server dry run) alertmanagerconfig.monitoring.coreos.com/platform-routing serverside-applied NAME AGE platform-routing 3s
Checkpoint: Route is loaded without exposing credentials
Continue whenAlertmanager selects the config, receiver Secret exists through approved delivery, routing tree is valid, and no literal credential appears in Git or command history.
Stop whenA broad route captures other teams, a credential is literal, or inhibition can hide unrelated pages.
If this step fails
AlertmanagerConfig is ignored.
Likely causeAlertmanager selectors, namespace selection, labels, API version, or RBAC do not include it.
kubectl -n {{monitoringNamespace}} get alertmanager main -o yamlkubectl -n {{monitoringNamespace}} describe alertmanagerconfig platform-routing
ResolutionAlign only the intended selector and namespace boundary and verify generated configuration.
Security notes
- Anyone able to read the receiver Secret can impersonate notifications or exfiltrate data. Restrict list/watch/get and Pod creation in the namespace.
Alternatives
- Use a provider-native identity integration when supported so Alertmanager does not hold a long-lived token.
Stop conditions
- Never paste a receiver URL into `--set` or Git.
verification
Exercise firing, delivery, grouping, and resolution
Create a controlled replica deficit in the canary, observe Pending and Firing state, confirm one test notification, restore replicas, and confirm resolved delivery within the expected latency.
Why this step matters
An alert is not operational until a real condition travels from scrape through rule evaluation, Alertmanager, receiver, human context, and resolution. Synthetic API injection skips important layers.
What to understand
Use only a disposable canary. Record start time, scrape interval, evaluation interval, pending duration, groupWait, receiver time, acknowledgement, recovery, and resolved time.
Observe that the expression first becomes true, the rule remains Pending for its `for` duration, then Firing reaches Alertmanager.
Confirm exactly one appropriately grouped notification with correct severity, owner, namespace, impact, value, and runbook.
Restore the prior replica count through the owning deployment workflow. Confirm Prometheus resolves, Alertmanager sends resolved, and no repeat remains.
Test receiver unavailability separately and confirm retry, queue, and failure alerts without flooding or losing unrelated routes.
System changes
- Temporarily scales the disposable canary to zero, fires an alert, then requires restoration to its recorded replica count.
Syntax explained
--replicas=0- Creates a controlled canary deficit; never use on a production workload for alert testing.
-w- Watches deployment state until interrupted; it does not restore replicas automatically.
Values stay on this page and are never sent or saved.
kubectl -n {{applicationNamespace}} scale deployment checkout --replicas=0 && kubectl -n {{applicationNamespace}} get deployment checkout -wdeployment.apps/checkout scaled NAME READY UP-TO-DATE AVAILABLE AGE checkout 0/0 0 0 12d Alert: DeploymentAvailabilityBelowTarget State: firing Receiver: platform-alert-canary Resolved: 2026-07-28T14:25:41Z
Checkpoint: One actionable page fires and resolves end to end
Continue whenRule transitions Pending to Firing, one correct notification arrives, runbook is accessible, restored replicas resolve the alert, and resolved notification arrives.
Stop whenThe target is production, duplicates flood, route is wrong, credential leaks, or resolution does not clear.
If this step fails
Alert does not resolve after replicas return.
Likely causeMetrics are stale, target is Down, expression labels changed, recording rule lags, or another deficit remains.
kubectl -n {{applicationNamespace}} get deploy,podkubectl -n {{monitoringNamespace}} get prometheusrule platform-workloads -o yaml
ResolutionTrace source metric and recorded series, restore target health, and correct alert identity before accepting the route.
Security notes
- Use a test receiver and sanitized labels; never page real incident channels with repeated uncontrolled drills.
Alternatives
- Use promtool rule fixtures for CI and retain this real canary for periodic end-to-end delivery validation.
Stop conditions
- Abort immediately if a non-canary workload is selected.
instruction
Provision Grafana dashboards and validate query meaning
Manage data sources and dashboards as reviewed code, secure access, validate time range and freshness, and ensure every panel states its operational question and missing-data behavior.
Why this step matters
Dashboards are shared operational interfaces. Provisioned definitions, stable queries, and access control prevent manual drift and misleading panels from becoming undocumented dependencies.
What to understand
Provision data-source URL, TLS, authentication, and dashboards through reviewed objects without literal credentials. Keep administrator identity external.
Each panel should answer one question: availability, latency, errors, saturation, capacity, rollout, queue, or dependency. State units and aggregation.
Validate current and historical time ranges, timezone, refresh, variables, label matchers, and missing-data display against known canary events.
Use recording rules for repeated expensive joins. Measure query duration and series fan-out before placing a panel on a wallboard.
Dashboards support investigation; alert rules remain separately tested code. A red panel is not automatically a page and a green panel is not proof of user success.
System changes
- No direct changes in this step; expects dashboards and data sources to be provisioned through the release workflow.
Syntax explained
127.0.0.1:3000- Binds the temporary port-forward locally instead of exposing Grafana publicly.
Values stay on this page and are never sent or saved.
kubectl -n {{monitoringNamespace}} get deploy,svc,configmap -l app.kubernetes.io/name=grafana && kubectl -n {{monitoringNamespace}} port-forward svc/grafana 3000:3000NAME READY UP-TO-DATE AVAILABLE deployment.apps/grafana 1/1 1 1 Forwarding from 127.0.0.1:3000 -> 3000
Checkpoint: Dashboards are reproducible, current, and meaningful
Continue whenAuthenticated users see provisioned dashboards whose panels match known canary events, units, time ranges, data freshness, and missing-data semantics.
Stop whenA dashboard requires shared admin credentials, manual edits, anonymous access, or an unexplained high-cost query.
If this step fails
Dashboard variable returns no namespaces.
Likely causeData source, label name, time range, query permissions, or provisioning differs from the dashboard assumption.
kubectl -n {{monitoringNamespace}} get configmap -l grafana_dashboard=1kubectl -n {{monitoringNamespace}} logs deploy/grafana --since=15m
ResolutionInspect the exact provisioned query and current label names, correct code, and redeploy rather than editing only the live UI.
Security notes
- Dashboard snapshots and share links can leak topology and metrics. Disable anonymous sharing unless explicitly approved.
Alternatives
- Use a managed Grafana instance with the same data-source, access, provisioning, and query-validation boundaries.
Stop conditions
- Do not publish dashboards with hard-coded credentials.
instruction
Operate storage, upgrades, rollback, and monitoring of monitoring
Alert on scrape, rule, notification, WAL, storage, Operator, and component health; rehearse failure; upgrade one supported release path at a time and keep last-known-good manifests and data.
Why this step matters
The monitoring system needs independent checks because it cannot reliably report its own total outage. Upgrades and retention changes are data and paging changes, not routine cosmetic maintenance.
What to understand
Monitor Prometheus target health, rule failures, evaluation duration, notifications, Alertmanager cluster, queue, Operator rejections, Pod restarts, PVC capacity, WAL, compaction, Grafana data-source errors, and kube-state-metrics health.
Use an external dead-man or canary path to detect total cluster or Alertmanager failure. Verify it without making one provider the only source of truth.
Back up provisioned code and Alertmanager/Grafana configuration through Git and secret recovery. Decide whether local Prometheus history is rebuilt, snapshotted, or retained remotely.
Read release notes and compatibility for every kube-prometheus, Operator, Prometheus, Alertmanager, Grafana, and kube-state-metrics transition. Test CRDs, targets, rules, routes, dashboards, and canary delivery.
Rollback generated manifests and component versions through the supported path while preserving PVCs and CRDs. Repeat discovery and end-to-end alert tests before closing change.
System changes
- No direct changes; establishes continuous health, capacity, recovery, upgrade, and rollback operations.
Syntax explained
Operator rejected resources- A critical self-monitoring signal that a monitor or rule object failed validation or selection.
external canary- A check outside the monitored cluster that can detect complete internal monitoring failure.
Values stay on this page and are never sent or saved.
kubectl -n {{monitoringNamespace}} get prometheus,alertmanager,servicemonitor,podmonitor,prometheusrule,alertmanagerconfig && kubectl -n {{monitoringNamespace}} get pod,pvc,pdb && kubectl -n {{monitoringNamespace}} get events --sort-by=.lastTimestampNAME VERSION REPLICAS READY prometheus.monitoring.coreos.com/k8s v3.12.0 2 2 alertmanager.monitoring.coreos.com/main v0.33.0 3 3 NAME STATUS VOLUME CAPACITY persistentvolumeclaim/prometheus-k8s-db Bound pvc-... 100Gi
Checkpoint: Monitoring failures are independently detectable and reversible
Continue whenFailure drills trigger external and internal signals, capacity has headroom, recovery artifacts exist, and pre/post-upgrade canaries pass.
Stop whenThe only alert path depends on the failed cluster, storage is near full, or rollback loses CRDs, configuration, or required data.
If this step fails
All dashboards disappear during an outage with no notification.
Likely causeMonitoring, dashboards, and receiver path share the same cluster and failure domain with no external canary.
kubectl -n {{monitoringNamespace}} get pod,svc,pvckubectl get node
ResolutionRestore the stack from pinned code and secret workflow, then implement an independent dead-man or external probe and test it.
Security notes
- Preserve alert and query evidence under incident access controls; monitoring data can expose operational weaknesses.
Alternatives
- Remote-write to an independent environment for longer retention and failure isolation after backpressure and identity are tested.
Stop conditions
- Do not upgrade when the canary route, last-known-good manifests, or PVC rollback boundary is unknown.
Finish line
Verification checklist
kubectl -n {{monitoringNamespace}} get deploy,statefulset,daemonset,pod -o wideAll expected components are Ready at pinned compatible versions and immutable image identities.kubectl -n {{applicationNamespace}} get service,servicemonitor,endpointslice --show-labelsThe intended ServiceMonitor selects only the expected ready endpoints, Prometheus reports them Up, and a negative selector fixture remains undiscovered.kubectl -n {{monitoringNamespace}} describe prometheusrule platform-workloadsRecording and alerting rules are loaded without rejection, pass fixtures, remain within query budgets, and contain owner, severity, impact, and runbook.kubectl -n {{monitoringNamespace}} get alertmanagerconfig platform-routing -o yamlThe canary moves Pending to Firing, delivers one correctly grouped notification, resolves after recovery, and exposes no receiver credential.kubectl -n {{monitoringNamespace}} get configmap -l grafana_dashboard=1Provisioned dashboards reproduce known canary events with correct time range, freshness, units, stable labels, and missing-data behavior.kubectl -n {{monitoringNamespace}} get prometheus,alertmanager,pod,pvc,pdbStorage has headroom, HA replicas and disruption policy match design, external canary detects total failure, and rollback artifacts are current.Recovery guidance
Common problems and safe checks
Prometheus Operator rejects a ServiceMonitor.
Likely causeThe object fails CRD validation, references invalid fields, or uses an API shape incompatible with the installed Operator.
kubectl -n {{applicationNamespace}} describe servicemonitorkubectl -n {{monitoringNamespace}} logs deploy/prometheus-operator --since=15m
ResolutionCorrect the manifest against the installed API reference and server-side dry run; never disable CRD validation to force it through.
A ServiceMonitor exists but no target appears.
Likely causePrometheus monitor selectors do not match it, its namespace selector excludes the application, or RBAC prevents discovery.
kubectl -n {{monitoringNamespace}} get prometheus -o yamlkubectl -n {{applicationNamespace}} get servicemonitor --show-labels
ResolutionAlign the smallest intended labels and namespace scope and grant only required discovery permission.
Target appears but is Down.
Likely causeService selectors, EndpointSlices, named port, path, scheme, TLS, authentication, NetworkPolicy, or application listener is wrong.
kubectl -n {{applicationNamespace}} get svc,endpointslice -o widekubectl -n {{applicationNamespace}} describe servicemonitor
ResolutionTrace Service to ready endpoint and test the metrics path from an authorized debug context; do not disable TLS verification as a permanent fix.
kube-state-metrics shows desired replicas but application traffic fails.
Likely causeObject-state metrics describe Kubernetes declarations and conditions, not end-to-end application correctness.
kubectl -n {{applicationNamespace}} get deploy,pod,svc,endpointslicekubectl -n {{monitoringNamespace}} get prometheusrule
ResolutionAdd application latency, error, correctness, queue, and synthetic signals; retain object-state alerts as supporting evidence.
Prometheus memory grows rapidly.
Likely causeUnbounded labels, excessive targets, short scrape intervals, broad kube-state-metrics allowlists, or expensive rules increased active series.
kubectl -n {{monitoringNamespace}} top podkubectl -n {{monitoringNamespace}} get prometheus -o yaml
ResolutionIdentify cardinality by metric and label, remove uncontrolled dimensions at instrumentation or relabeling, and resize only after the cause is bounded.
PrometheusRule exists but the alert never becomes Pending.
Likely causeThe selector excludes the rule, expression returns no series, labels differ, data is stale, or rule validation failed.
kubectl -n {{monitoringNamespace}} describe prometheusrulekubectl -n {{monitoringNamespace}} logs deploy/prometheus-operator --since=15m
ResolutionRun the exact expression for the intended interval, inspect rule status and selectors, and add fixture-based rule tests.
Alert remains Pending and never fires.
Likely causeThe condition clears before the `for` duration, evaluation has gaps, or labels create discontinuous series.
kubectl -n {{monitoringNamespace}} get prometheusrule -o yamlkubectl -n {{monitoringNamespace}} get pod -l app.kubernetes.io/name=prometheus
ResolutionConfirm the desired persistence and series continuity. Change `for` only from SLO and noise evidence, not to force a demonstration.
Alert fires in Prometheus but not in Alertmanager.
Likely causePrometheus alertmanager discovery, Service name, port, TLS, NetworkPolicy, or generated configuration is wrong.
kubectl -n {{monitoringNamespace}} get prometheus,alertmanager,svc,endpointslicekubectl -n {{monitoringNamespace}} logs prometheus-k8s-0 --since=15m
ResolutionRepair the Operator-managed alertmanager reference and connectivity, then repeat the canary from Pending through resolved notification.
Alert reaches Alertmanager but no notification arrives.
Likely causeRoute matchers, receiver credential, template, inhibition, silence, egress, or downstream API is wrong.
kubectl -n {{monitoringNamespace}} get alertmanagerconfig -o yamlkubectl -n {{monitoringNamespace}} logs alertmanager-main-0 --since=15m
ResolutionInspect sanitized routing status and receiver errors, repair the narrow route or secret reference, and test both firing and resolved delivery.
Hundreds of duplicate pages arrive.
Likely causeGrouping labels are too granular, HA replicas are not clustered correctly, rules emit unstable labels, or receiver retries are misconfigured.
kubectl -n {{monitoringNamespace}} get alertmanager -o yamlkubectl -n {{monitoringNamespace}} get prometheusrule -o yaml
ResolutionStabilize alert identity, group by actionable dimensions, verify HA peer state, and tune intervals against the incident workflow.
Grafana panel says No data while Prometheus query works.
Likely causeWrong data source, time range, dashboard variable, label matcher, tenant header, refresh interval, or query transformation.
kubectl -n {{monitoringNamespace}} get configmap -l grafana_dashboard=1kubectl -n {{monitoringNamespace}} logs deploy/grafana --since=15m
ResolutionUse query inspector with a non-sensitive account, compare the exact PromQL and time range, and fix provisioned dashboard code.
Prometheus persistent volume fills.
Likely causeRetention time and size, ingestion, WAL, compaction, cardinality, or remote-write backlog exceed capacity.
kubectl -n {{monitoringNamespace}} get pvckubectl -n {{monitoringNamespace}} top pod
ResolutionProtect availability, reduce uncontrolled ingestion, set tested time and size retention, expand storage through the supported class, and verify historical-query requirements.
Upgrade leaves CRDs or rules incompatible.
Likely causeThe release branch, Kubernetes version, CRD migration, removed field, dashboard, or rule was not tested through the supported upgrade path.
kubectl -n {{monitoringNamespace}} get podkubectl get crd | grep monitoring.coreos.com
ResolutionStop changes, preserve Prometheus data and existing CRDs, restore the last supported manifests, and repeat target and alert canaries in a representative cluster.
Recovery
Rollback
Revert only the most recent monitoring manifest, rule, selector, route, dashboard, retention, or component release through the pinned source workflow; preserve CRDs, PVCs, historical data, evidence, and the last working alert route while proving target and notification canaries.
- Stop the rollout and capture sanitized Operator events, rejected resources, generated-object status, target health, rule health, Alertmanager status, component versions, image IDs, PVC capacity, and canary timestamps.
- For a bad ServiceMonitor, PrometheusRule, AlertmanagerConfig, or dashboard, revert the exact source revision; do not broaden selectors, disable validation, expose endpoints, or paste temporary credentials.
- For component regression, restore the last supported kube-prometheus release and documented CRDs through the official upgrade or rollback path while preserving Prometheus and Alertmanager PVCs.
- Wait for Operator, Prometheus, Alertmanager, kube-state-metrics, Grafana, and node-exporter readiness; reconcile expected target count and rule loading.
- Repeat application target discovery, negative discovery, rule fixture, Pending, Firing, grouped delivery, resolved delivery, dashboard, missing-data, and external-canary checks.
- If a credential was exposed, rotate it through the secret system, inspect access logs, and verify the old credential no longer works without putting either value in evidence.
- Resume change only after owners accept query performance, cardinality, storage headroom, routing, and end-to-end alert evidence.
Evidence