Build actionable Prometheus alerts and Alertmanager routing
Create tested Prometheus 3.13.1 rules and an Alertmanager 0.32.1 HA route tree with stable labels, grouping, deduplication, inhibition, protected secrets and a safe nonpaging test lane.
Deliver fewer, actionable and resilient notifications whose rule behavior, routing, suppression and failure modes are tested without ever using production paging as the experiment.
- Prometheus 3.13.1
- Alertmanager 0.32.1
- promtool same patch as Prometheus
- amtool same release as Alertmanager
- Ownership and SLO Service owners, impact/severity definitions, escalation policy and maintained runbook URLs.
- Metric semantics Stable metric/label contracts, known scrape/evaluation intervals and representative historical/fixture data.
- HA hosts and network Three failure domains, stable private peer addresses, cluster mesh connectivity and protected web/API paths.
- Receiver safety Physically nonpaging test receiver plus monitored production catch-all and audited integration credentials.
- Release pipeline Signed versioned rules/config, secret injection, promtool/amtool tests and staged canary reload.
- Meta-monitoring Independent visibility into rule evaluation, notification send/queue, Alertmanager peers, receiver errors and safe canary latency.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- Source-controlled Prometheus 3.13.1 alert and recording rules with stable ownership labels, actionable annotations, evidence-based `for` durations, absence/low-volume handling and promtool unit tests for firing and non-firing cases.
- A three-peer Alertmanager 0.32.1 cluster with consistent signed configuration, deliberate grouping/deduplication, narrowly scoped inhibition, protected receiver secrets and Prometheus configured to send to every peer.
- A safe synthetic routing lane that can validate receiver selection, grouping, inhibition, resolution and HA failure without paging production, plus dashboards and rollback evidence for the alerting system itself.
- Every page names the affected service/environment, states observable impact, links an owned runbook, supplies useful summary/description and avoids volatile alert labels.
- Synthetic labels route only to the safe test receiver; critical alerts inhibit only related warnings; grouped notifications remain actionable and duplicate behavior during peer failure is known.
- Rules and Alertmanager configuration pass static and unit tests, all peers run identical digests, secrets stay outside repositories/output, and notification delivery is monitored end to end.
Architecture
How the parts fit together
Prometheus 3.13.1 evaluates versioned rule groups at a known interval. Recording rules precompute expensive stable expressions; alert rules describe user or service symptoms and carry stable `team`, `service`, `environment`, `severity` and `runbook_url` metadata. `promtool check rules` validates syntax and `promtool test rules` evaluates fixtures including counter resets, absent series, low traffic and sustained thresholds. Each Prometheus sends alerts to all three Alertmanager 0.32.1 peers. Alertmanager peers share silence and notification-log state through their cluster mesh, but configuration is deployed identically by the release system. The route tree groups related alerts, directs explicitly labeled synthetic tests to a nonpaging receiver and uses narrow inhibition so a critical root cause suppresses only its matching warnings. Receiver credentials are protected files or deployment-injected secrets. Delivery and cluster health are monitored independently, and rollback restores the last validated rules/config rather than silencing unknown failures.
- Select a symptom and owner, explore historical data, define stable labels and an actionable response before writing a rule.
- Implement recording/alert expressions and annotations, then unit-test normal, sustained failure, low traffic, absence and reset cases.
- Validate and canary-reload Prometheus rules; confirm pending-to-firing-to-resolved behavior in its API.
- Route synthetic alerts through a nonpaging lane, prove grouping and inhibition, then stage the same signed configuration across all Alertmanager peers.
- Prometheus sends each alert to every peer; cluster state deduplicates healthy-path notification, while partitions may cause bounded duplicates.
- Operators monitor rule, send, peer and receiver health; rollback restores known-good config without disabling the alerting boundary.
Assumptions
- Prometheus 3.13.1 is the current reviewed patch identified by the official download catalog in July 2026. Alertmanager 0.32.1 is the pinned current production release for this guide.
- Three Alertmanager peers run across independent failure domains with stable private addresses, TLS/authenticated operator access and cluster mesh connectivity. Two peers may be accepted for a smaller risk profile, but one is not HA.
- Each Prometheus instance is configured to send to every Alertmanager peer rather than a load balancer that selects one target.
- Service owners provide SLO/impact, severity policy, runbook and escalation. Monitoring operators do not invent paging policy from a metric name alone.
- Metrics have stable semantic labels; customer identifiers, request IDs, pod UIDs and other volatile values are never alert labels.
- Receiver integrations offer a nonproduction endpoint/channel with no production paging side effect. Test credentials and labels are distinct.
- Secrets are injected by an approved mechanism, protected by filesystem/service identity and excluded from source control, rendered config output and backups.
- Time is synchronized, Prometheus evaluation/scrape intervals are known, and rule evaluation capacity has headroom.
- Silences are time-bound operational controls with owner and change reference, not a substitute for fixing a noisy or incorrect rule.
Key concepts
- Pending and `for`
- An alert remains pending until its expression is continuously true for the configured duration. Missing evaluations or changing label sets can reset pending state.
- Grouping
- Alertmanager combines alerts sharing selected labels into notifications, reducing fan-out while preserving an operator-sized incident boundary.
- Deduplication
- Alertmanager notification-log state suppresses repeated equivalent notifications according to group and repeat timing. Network partitions can still create duplicates.
- Inhibition
- A firing source alert suppresses matching target alerts. Empty and missing labels can compare as equal, so labels and matchers must be deliberate.
- Silence
- A time-bounded matcher set that mutes notifications. It should have an owner and explanation and does not resolve the underlying alert.
- Alert fingerprint
- Identity derived from alert labels. Volatile labels create new alert instances and defeat stable pending, grouping and deduplication.
- Route inheritance
- Child routes inherit unset receiver/timing/group properties from parents. `continue` can route the same alert to later siblings and must be explicit.
- HA mesh
- Alertmanager peer communication that shares silences and notification state. It does not distribute configuration and does not guarantee zero duplicates during partitions.
Before you copy
Values used in this guide
{{prometheusVersion}}Approved Prometheus 3.13.1 release.
Example: 3.13.1{{alertmanagerVersion}}Pinned Alertmanager release.
Example: 0.32.1{{rulesPath}}Versioned rule directory.
Example: /etc/prometheus/rules{{alertmanagerConfig}}Rendered protected Alertmanager configuration.
Example: /etc/alertmanager/alertmanager.yml{{alertmanagerUrl}}Private operator/API URL for a test peer.
Example: https://alertmanager-01.internal.example{{testReceiverUrlFile}}secretProtected file containing nonpaging webhook URL.
Example: /etc/alertmanager/secrets/test-webhook-url{{triageReceiverUrlFile}}secretProtected file containing monitored catch-all webhook URL.
Example: /etc/alertmanager/secrets/triage-webhook-url{{pagerReceiverUrlFile}}secretProtected file containing production pager webhook URL.
Example: /etc/alertmanager/secrets/payments-pager-webhook-url{{runbookBase}}Owned runbook URL base.
Example: https://oneliners.guru/runbooks{{testAlertId}}Unique nonsecret synthetic alert identifier.
Example: ol-alert-20260728-001Security and production boundaries
- Alert annotations can contain label values and links that reach third-party receivers. Never include secrets, raw customer payloads or unrestricted URLs; sanitize external labels.
- Receiver credentials authorize paging, chat, email or webhook actions and can be abused for spam or data exfiltration. Use least privilege, protected injection, rotation and audit.
- Prometheus and Alertmanager APIs expose topology, alerts and operational state. Protect them with network and authentication controls; do not publish admin/reload endpoints.
- A test route must be incapable of production paging even when labels are malformed. Route preview and a monitored catch-all prevent silent or dangerous fallthrough.
- Silences and inhibition can hide incidents. Audit creation/config changes and alert on unexpectedly broad or long-lived suppression.
Stop before continuing if
- Stop if a synthetic label can reach a production pager, route preview differs from delivery, the default route silently drops unmatched production alerts or receiver secrets appear in rendered output.
- Stop if promtool tests omit negative/low-volume/absence cases, volatile labels define alert identity, or the runbook/owner is missing.
- Stop if Prometheus sends to only one HA peer, peer mesh is unhealthy, configuration digests differ or operator APIs are public.
- Stop if inhibition hides unrelated alerts, broad silences exist without owner/expiry, or test notification cannot be traced end to end.
- Stop rollout on evaluation failures, notification errors, duplicate storm, missing resolve notifications or degraded paging SLO.
decision
Define symptom, owner, severity and response contract
For each proposed alert, document user/service impact, evidence, owner, runbook, urgency, minimum actionable duration and what the responder can do. Prefer symptoms over component internals and stable labels over volatile instances. Decide whether absence is failure or expected inactivity.
Why this step matters
An alert is an operational contract, not a threshold. If no owner can act from its evidence, notification adds noise and erodes trust.
What to understand
Use severity definitions tied to response time and impact, not adjectives.
Keep routing labels stable; put pod/instance/request detail in annotations or dashboards.
Explicitly decide low-traffic, maintenance and missing-series behavior.
System changes
- Creates alert design documentation; no system state changes.
Syntax explained
stable identity- Labels that keep one incident identity across ephemeral target changes.
Alert: ApiHighErrorBudgetBurn Impact: requests fail above SLO budget Owner: payments-platform Severity: critical Stable identity: service, environment, cluster Runbook: /runbooks/http-5xx Minimum volume: 10 requests/s Immediate operator action: validate dependencies and rollback recent release
Checkpoint: Verify this layer before continuing
Continue whenEach alert has owned impact, severity, stable labels, runbook, action and edge-case policy.
Stop whenStop if no action/owner exists, labels are volatile, or severity is unsupported by impact.
If this step fails
Rule design requires `pod` as an alert label.
Likely causeThe alert describes an instance metric instead of a service symptom.
Review operator decision boundary.Determine whether pod belongs only in annotation.
ResolutionAggregate to stable service/cluster identity and link per-instance investigation.
Security notes
- Do not place customer identifiers or raw request data in labels/annotations.
Alternatives
- Use a dashboard/SLO report instead of paging when no immediate action exists.
Stop conditions
- Stop if no action/owner exists, labels are volatile, or severity is unsupported by impact.
command
Verify pinned Prometheus and Alertmanager artifacts
Pin Prometheus 3.13.1 and Alertmanager 0.32.1 from official signed/checksummed releases, record immutable digests and review release/security notes. Verify the service binaries, promtool and amtool share expected versions.
Why this step matters
Rule language, validation and Alertmanager configuration behavior are versioned. Exact binary evidence prevents testing with one tool and running another.
What to understand
Pin artifacts by digest/checksum and retain provenance plus vulnerability disposition.
Use the promtool/amtool bundled with deployed releases.
Review breaking changes before every minor upgrade and stage rule evaluation performance.
System changes
- Records artifact identity; no services change.
Syntax explained
--version- Prints build identity of the exact executable used by validation and service.
prometheus --version && promtool --version && alertmanager --version && amtool --versionprometheus, version 3.13.1 promtool, version 3.13.1 alertmanager, version 0.32.1 amtool, version 0.32.1
Checkpoint: Verify this layer before continuing
Continue whenAll four binaries match reviewed immutable releases and support the intended syntax.
Stop whenStop on floating/unverifiable artifact, version mismatch or unresolved security advisory.
If this step fails
promtool reports 3.12 while Prometheus is 3.13.
Likely causePATH points to a workstation/legacy tool.
Inspect service paths.Resolve each executable and checksum.
ResolutionUse absolute paths to bundled reviewed tools and remove ambiguity through change control.
Security notes
- Do not execute release assets before checksum/signature verification.
Alternatives
- Use an approved mirrored artifact repository.
Stop conditions
- Stop on floating/unverifiable artifact, version mismatch or unresolved security advisory.
config
Write recording and alert rules with stable metadata
Create a recording rule for request rate/error rate and an alert requiring both SLO burn and minimum traffic. Add stable team, service, environment and severity labels plus concise summary, description and runbook URL. Use `for` and `keep_firing_for` only from documented incident behavior.
Why this step matters
Recording rules make reused expressions explicit and reduce expensive repeated work. Minimum volume prevents meaningless ratios, while stable labels preserve alert identity.
What to understand
Align numerator/denominator dimensions and range. Use `rate` for counters and test resets.
`for` filters transient events; `keep_firing_for` can bridge brief data gaps but must not conceal recovery.
Template only approved label values into annotations and ensure runbook links are controlled.
System changes
- Adds rule evaluation load and a potential page after sustained production conditions.
Syntax explained
sum by (...)- Aggregates ephemeral instances into the stable incident boundary.
and on (...)- Applies the minimum-traffic condition using the exact stable label match.
for: 10m- Requires continuous truth for ten minutes before firing.
{{rulesPath}}/api-alerts.ymlValues stay on this page and are never sent or saved.
groups:
- name: api-slo
interval: 30s
rules:
- record: service:http_requests:rate5m
expr: sum by (service, environment, cluster) (rate(http_requests_total[5m]))
- alert: ApiHighErrorRate
expr: |
(
sum by (service, environment, cluster) (rate(http_requests_total{code=~"5.."}[5m]))
/
sum by (service, environment, cluster) (rate(http_requests_total[5m]))
) > 0.05
and on (service, environment, cluster)
service:http_requests:rate5m > 10
for: 10m
keep_firing_for: 5m
labels:
severity: critical
team: payments-platform
annotations:
summary: "High API error rate for {{ $labels.service }}"
description: "5xx ratio stayed above 5% for 10 minutes with more than 10 requests/s."
runbook_url: "{{runbookBase}}/http-5xx"Checking /etc/prometheus/rules/api-alerts.yml SUCCESS: 2 rules found Rule labels: environment, service, severity, team Volatile labels: 0 Runbook URL: PASS
Checkpoint: Verify this layer before continuing
Continue whenpromtool validates, stable labels are bounded and expression behaves correctly on historical normal/failure periods.
Stop whenStop if vector matching is ambiguous, low traffic pages, labels are volatile or annotations expose sensitive values.
If this step fails
Rule produces many alerts, one per pod.
Likely causeAggregation retained pod/instance in the label set.
Inspect expression output labels.Compare intended incident grouping.
ResolutionAggregate by stable service/environment/cluster and preserve instances in query links instead.
Security notes
- Annotations can leave the monitoring boundary through receivers; treat them as externally visible.
Alternatives
- Use a dashboard-only recording rule until paging action and owner are ready.
Stop conditions
- Stop if vector matching is ambiguous, low traffic pages, labels are volatile or annotations expose sensitive values.
verification
Unit-test firing, non-firing, low traffic and missing data
Create promtool fixtures for healthy traffic, sustained failure, short spike, low denominator, counter reset and absent series. Assert alert labels and annotations at exact evaluation times. A syntax check without behavioral fixtures is insufficient.
Why this step matters
PromQL often fails at edge cases that are invisible in a happy-path graph. Deterministic fixtures protect behavior through refactoring and upgrades.
What to understand
Use realistic evaluation and scrape intervals. Assert the `for` transition at boundaries.
Include no-fire tests; a suite that only proves firing cannot detect noise.
Model counter reset and missing series explicitly instead of smoothing them away.
System changes
- Reads rule/test files and executes an offline evaluator; no Prometheus reload occurs.
Syntax explained
check rules- Validates rule syntax and structure.
test rules- Evaluates declared series and expected alerts at deterministic times.
Values stay on this page and are never sent or saved.
promtool check rules {{rulesPath}}/*.yml && promtool test rules {{rulesPath}}/tests/*.test.ymlChecking /etc/prometheus/rules/api-alerts.yml SUCCESS: 2 rules found Unit Testing: /etc/prometheus/rules/tests/api-alerts.test.yml api healthy: SUCCESS sustained 5xx: SUCCESS short spike: SUCCESS low traffic: SUCCESS counter reset: SUCCESS absent input: SUCCESS
Checkpoint: Verify this layer before continuing
Continue whenAll positive/negative edge fixtures pass with exact stable labels and annotations.
Stop whenStop if tests omit low traffic, absence, reset, pending duration or expected no-fire behavior.
If this step fails
Counter reset fixture fires unexpectedly.
Likely causeExpression used raw increase/rate incorrectly or fixture range is too short.
Evaluate numerator/denominator separately.Inspect fixture samples around reset.
ResolutionCorrect counter semantics/range and retain the reset fixture as regression coverage.
Security notes
- Fixtures use synthetic values and contain no production customer labels.
Alternatives
- Use promtool in CI and a canary Prometheus for additional integration coverage.
Stop conditions
- Stop if tests omit low traffic, absence, reset, pending duration or expected no-fire behavior.
config
Design a safe route tree and timing policy
Make route ownership explicit. Put a synthetic `notification_class=test` route first and send it to a physically nonpaging receiver. Use a monitored catch-all for unmatched production alerts. Set grouping and timing per operational boundary; review inheritance and `continue` carefully.
Why this step matters
A route tree is executable escalation policy. Safe defaults and a nonpaging test lane allow full integration tests without relying on fragile silences.
What to understand
Group by stable incident boundaries, not every label and not so broadly that independent environments merge.
group_wait permits related alerts to arrive; group_interval controls updates; repeat_interval controls reminders.
A monitored catch-all exposes ownership gaps. A null default can silently lose real pages.
System changes
- Changes notification recipients/timing and can page or suppress incidents if incorrect.
Syntax explained
continue: false- Prevents a matched synthetic alert from continuing into later production routes.
group_by- Defines which alert labels form one notification group.
url_file- Reads a protected receiver URL from a file instead of embedding it in YAML.
{{alertmanagerConfig}}Values stay on this page and are never sent or saved.
route:
receiver: triage-catchall
group_by: [alertname, service, environment, cluster]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- receiver: safe-test-webhook
matchers: ['notification_class="test"']
continue: false
- receiver: pager-payments
matchers: ['team="payments-platform"', 'severity="critical"']
receivers:
- name: safe-test-webhook
webhook_configs:
- url_file: {{testReceiverUrlFile}}
- name: triage-catchall
webhook_configs:
- url_file: {{triageReceiverUrlFile}}
- name: pager-payments
webhook_configs:
- url_file: {{pagerReceiverUrlFile}}Routing tree: test-route -> safe-test-webhook (continue=false) critical/payments -> pager-payments warning/payments -> chat-payments unmatched -> triage-catchall Route test notification_class=test: safe-test-webhook Production pager side effects: 0
Checkpoint: Verify this layer before continuing
Continue whenEvery approved label set has the expected receiver/timing; synthetic route cannot page and unmatched production alerts are visible.
Stop whenStop if test alert can fall through, catch-all discards silently, grouping merges unrelated incidents or secret is embedded.
If this step fails
Route test selects both safe and pager receivers.
Likely cause`continue` is true or another sibling also matches after the test route.
Run amtool route test with exact labels.Inspect route order and inherited continue.
ResolutionPut the exact test route first, set continue false and repeat preview before any API alert.
Security notes
- Receiver URLs/tokens remain protected and route outputs must not print them.
Alternatives
- Use a dedicated nonproduction Alertmanager for integration tests, still keeping a safe production synthetic lane.
Stop conditions
- Stop if test alert can fall through, catch-all discards silently, grouping merges unrelated incidents or secret is embedded.
config
Add narrow inhibition for root-cause alerts
Inhibit warning alerts only when a related critical alert with the same alertname/service/environment/cluster (or a deliberately paired cause label) is firing. Ensure every `equal` label is present and nonempty on both source and target. Unit-test related, unrelated and missing-label cases.
Why this step matters
Inhibition reduces symptom noise during a root cause but can hide unrelated failures. Exact shared labels and negative fixtures contain that risk.
What to understand
Missing and empty labels can compare equally. Require labels in rule review and test fixtures.
Use a distinct cause relationship when critical/warning alert names differ; do not broaden to all warnings in a cluster.
Inhibition affects notification, not alert visibility; operators should still see inhibited alerts in UI/API.
System changes
- Suppresses notifications for matching warnings while the source critical alert fires.
Syntax explained
source_matchers- Selects the root-cause/higher-severity firing alert.
target_matchers- Selects candidate symptom/lower-severity alerts.
equal- Requires exact identity match on every listed label, including empty/missing semantics.
{{alertmanagerConfig}}inhibit_rules:
- source_matchers:
- severity="critical"
target_matchers:
- severity="warning"
equal:
- alertname
- service
- environment
- clustercritical + related warning: warning inhibited critical + different service: warning delivered critical + different environment: warning delivered missing service label: fixture rejected critical resolved: warning delivered
Checkpoint: Verify this layer before continuing
Continue whenOnly related warnings are inhibited and they reappear after critical resolves.
Stop whenStop if unrelated/missing-label warnings are hidden or source alert inhibits itself unexpectedly.
If this step fails
All warnings in an environment disappear.
Likely causeequal labels are insufficient or rule labels are missing.
Compare exact alert label sets.Test a different service and missing-label alert.
ResolutionNormalize required labels and narrow equal/matchers; remove inhibition until negative tests pass.
Security notes
- Audit inhibition changes because they can materially reduce incident visibility.
Alternatives
- Group related alerts without inhibition if root-cause relationship is uncertain.
Stop conditions
- Stop if unrelated/missing-label warnings are hidden or source alert inhibits itself unexpectedly.
config
Deploy a consistent three-peer Alertmanager cluster
Run three Alertmanager 0.32.1 peers with stable private cluster addresses across failure domains. Protect web/API access, persist local state where appropriate and deploy the same signed configuration digest to every peer. Remember gossip shares silences/notification state, not configuration.
Why this step matters
Alertmanager HA depends on direct peer state exchange and identical configuration. A load-balanced façade or gossip does not fix divergent route trees.
What to understand
Use stable advertise/listen addresses and allow only cluster peers on mesh ports.
Roll one peer at a time and verify full membership plus config digest after each reload.
Expect possible duplicate notification during partition and test/monitor it.
System changes
- Starts/reconfigures a notification cluster; divergence can duplicate or misroute pages.
Syntax explained
config show- Retrieves effective configuration for protected digest comparison; output can be sensitive.
sha256sum- Compares exact effective config across peers without treating digest as authenticity by itself.
Values stay on this page and are never sent or saved.
amtool --alertmanager.url={{alertmanagerUrl}} config show | sha256sumpeer alertmanager-01: digest 5ce9854b... peers 3 peer alertmanager-02: digest 5ce9854b... peers 3 peer alertmanager-03: digest 5ce9854b... peers 3 cluster status: ready public web/API reachability: denied
Checkpoint: Verify this layer before continuing
Continue whenAll peers show the same protected config digest, full mesh and private authenticated API.
Stop whenStop if digest differs, peer count is incomplete, mesh is public/untrusted or rolling update risks all replicas.
If this step fails
One peer remains alone in its cluster.
Likely causeAdvertise address or mesh firewall/NAT prevents peer communication.
Inspect peer metrics/status.Test mesh TCP/UDP paths privately.
ResolutionCorrect stable peer addressing/firewall and rejoin before proceeding to the next replica.
Security notes
- Effective config may contain receiver metadata; restrict `config show` and never log secrets.
Alternatives
- Use managed Alertmanager HA where operating peer mesh is unsupported.
Stop conditions
- Stop if digest differs, peer count is incomplete, mesh is public/untrusted or rolling update risks all replicas.
config
Send each Prometheus to every Alertmanager peer
Configure Prometheus alerting targets so each server sends alerts to all Alertmanager peers. Do not rely on a load balancer that chooses one peer. Validate service discovery and TLS/auth for every target, then stop one test peer and confirm delivery continues through the others.
Why this step matters
Prometheus must reach all peers so failure of one target does not discard alerts. Alertmanager clustering deduplicates the healthy path.
What to understand
Verify each target independently, including trust and authentication.
Monitor Prometheus notification queue, dropped/sent errors and discovered target count.
Use a nonpaging fixture during peer failure; partitions may produce duplicates and should be recorded.
System changes
- Changes alert delivery targets and can duplicate or lose notifications if configured incorrectly.
Syntax explained
static_configs.targets- Lists every HA peer so Prometheus sends to all rather than one load-balanced endpoint.
scheme: https- Uses TLS; add the approved HTTP client identity/trust configuration in the rendered deployment.
/etc/prometheus/prometheus.ymlalerting:
alertmanagers:
- static_configs:
- targets:
- alertmanager-01.internal.example:9093
- alertmanager-02.internal.example:9093
- alertmanager-03.internal.example:9093
scheme: httpsDiscovered Alertmanagers: alertmanager-01.internal.example:9093 UP alertmanager-02.internal.example:9093 UP alertmanager-03.internal.example:9093 UP Synthetic alert with peer 01 stopped: delivered by cluster: PASS production pages: 0
Checkpoint: Verify this layer before continuing
Continue whenEvery peer is discovered/authenticated and safe alert delivers during one-peer loss without production page.
Stop whenStop if only one endpoint is visible, TLS is disabled, notification queue drops or safe test falls into production route.
If this step fails
Prometheus reports only one Alertmanager target.
Likely causeService discovery/LB collapses peer endpoints into one address.
Inspect effective alerting config.Query discovered Alertmanager targets.
ResolutionExpose/configure all peer addresses according to supported HA pattern and repeat failure test.
Security notes
- Protect Prometheus-to-Alertmanager credentials and operator APIs.
Alternatives
- Use DNS/service discovery that returns every peer and prove Prometheus retains all targets.
Stop conditions
- Stop if only one endpoint is visible, TLS is disabled, notification queue drops or safe test falls into production route.
verification
Test route, grouping and inhibition without paging
First run `amtool config routes test` against the candidate. Then post synthetic alerts carrying `notification_class=test` and a unique ID to the protected v2 API. Exercise warning, related critical, unrelated warning, resolve and grouped alerts. Confirm only the safe receiver records expected messages.
Why this step matters
Static route preview proves matcher selection; API fixtures additionally prove grouping, inhibition, receiver authentication and resolved delivery. The nonpaging invariant is more reliable than silencing a production route.
What to understand
Use a dedicated staging environment label, unique ID and short synthetic duration.
Test exact related/unrelated label combinations and wait through group timing.
Query/audit receiver results and verify zero production integration calls.
System changes
- Creates and resolves synthetic alerts in Alertmanager and sends notifications only to the safe receiver.
Syntax explained
config routes test- Evaluates route selection without submitting an alert.
notification_class=test- Selects the physically nonpaging route that must stop further matching.
Values stay on this page and are never sent or saved.
amtool --alertmanager.url={{alertmanagerUrl}} config routes test notification_class=test team=payments-platform severity=critical service=checkout environment=staging cluster=eu-testRouting tree: matched receiver: safe-test-webhook continue: false Synthetic delivery: grouped warning count: 2 related warning inhibited by critical: PASS unrelated warning delivered: PASS resolved notification: PASS production paging events: 0
Checkpoint: Verify this layer before continuing
Continue whenPreview and delivery agree; grouping/inhibition/resolution pass and production receives nothing.
Stop whenStop before POST if preview shows pager, continue true, unknown catch-all or secret output.
If this step fails
Preview is safe but API fixture pages production.
Likely causeRunning config differs from candidate, another peer has divergent config or test route continues.
Compare effective config digest on all peers.Inspect notification audit and route order.
ResolutionDisable fixture, restore consistent safe config and repeat preview plus one alert.
Security notes
- Synthetic annotations contain no customer data, secrets or uncontrolled links.
Alternatives
- Use a dedicated isolated Alertmanager to validate candidate before safe-lane production test.
Stop conditions
- Stop before POST if preview shows pager, continue true, unknown catch-all or secret output.
verification
Canary reload rules and observe lifecycle
Load tested rules into a canary Prometheus, inspect `/api/v1/rules`, generate only synthetic source metrics and observe inactive → pending → firing → resolved timing. Then stage rule reload across production instances and confirm evaluation duration, failures and Alertmanager sends.
Why this step matters
Offline tests cannot prove rule loading, interval timing, state transitions or notification delivery. A canary prevents a malformed or expensive group from affecting every evaluator.
What to understand
Protect reload/API endpoints and use the deployment controller rather than public access.
Verify exact loaded rule, health, last error and evaluation duration.
Use synthetic source metrics routed to the safe receiver; do not manipulate production application metrics.
System changes
- Reloads rule configuration and creates synthetic alert lifecycle; a bad expression can add evaluation load or pages if routing is wrong.
Syntax explained
POST /-/reload- Requests supported configuration reload; endpoint must be protected.
/api/v1/rules- Shows loaded groups, evaluation health and alert state for verification.
curl --fail --silent --request POST https://prometheus-canary.internal.example/-/reload && curl --fail --silent https://prometheus-canary.internal.example/api/v1/rulesreload: HTTP 200 rule health: ok last evaluation: 2026-07-28T13:00:00Z state timeline: inactive -> pending (10m00s) -> firing -> resolved evaluation failures: 0 evaluation p95: 18ms safe receiver notifications: firing=1 resolved=1
Checkpoint: Verify this layer before continuing
Continue whenCanary lifecycle/timing and safe notifications match fixtures; production staged reload has zero failures and acceptable duration.
Stop whenStop if API is public, loaded expression differs, evaluation fails/slows, state timing is wrong or production receiver is selected.
If this step fails
Rule stays pending longer than expected.
Likely causeEvaluation gaps, changing labels or `for` duration/interval assumptions reset state.
Inspect alert activeAt and evaluation timestamps.Compare expression output labels over time.
ResolutionStabilize labels and evaluation health, correct duration assumptions and repeat canary.
Security notes
- Rule APIs reveal topology and labels; restrict access and sanitize evidence.
Alternatives
- Restart one canary instance with validated config if reload is not approved, preserving HA.
Stop conditions
- Stop if API is public, loaded expression differs, evaluation fails/slows, state timing is wrong or production receiver is selected.
warning
Monitor delivery, suppression, peers and rollback evidence
Alert on Prometheus rule/notification failures, queue drops, Alertmanager peer count, notification errors, unmatched production routes, broad silences and safe canary latency. Store signed secret-free rules/config digests and route-test outputs. Rehearse one-peer failure, partition duplicate behavior and restoration of the last validated revision.
Why this step matters
The alerting system is itself a production dependency. Meta-monitoring must reveal rule failure, delivery failure and excessive suppression without depending entirely on the same broken path.
What to understand
Use an external or independently routed heartbeat for total Alertmanager failure.
Audit silences by owner, comment, matcher breadth and expiry; inhibition changes remain code-reviewed.
Preserve secret-free configuration and route evidence; HA state does not distribute configuration or replace change history.
System changes
- Adds monitoring/alerts and recovery artifacts; failure exercises may create bounded duplicate test notifications.
Syntax explained
cluster show- Displays peer membership/health for the selected protected Alertmanager.
silence query- Lists active silences for breadth/ownership/expiry review.
Values stay on this page and are never sent or saved.
amtool --alertmanager.url={{alertmanagerUrl}} cluster show && amtool --alertmanager.url={{alertmanagerUrl}} silence queryPeers: alertmanager-01 ready alertmanager-02 ready alertmanager-03 ready Silences: 1 owner=maintenance@example expires=2026-07-28T15:00:00Z comment="CHG-1842" Safe canary delivery p95: 1.8s Unmatched production alerts: 0 Notification errors: 0
Checkpoint: Verify this layer before continuing
Continue whenAll peers/config digests match, canary delivery meets SLO, no unexplained suppression/errors exist and rollback drill passes.
Stop whenStop if meta-alerts depend solely on broken path, broad silence lacks owner/expiry, peer/config divergence exists or secrets enter backups.
If this step fails
All peers are healthy but safe canary is missing.
Likely causeReceiver credential/integration failed while cluster health remained green.
Inspect notification errors and receiver audit.Route-test exact canary labels.
ResolutionRepair/rotate the safe receiver identity and add delivery canary alert independent from peer liveness.
Security notes
- Operational reports must omit receiver credentials and sensitive alert annotations.
Alternatives
- Use an external black-box notification canary in addition to internal metrics.
Stop conditions
- Stop if meta-alerts depend solely on broken path, broad silence lacks owner/expiry, peer/config divergence exists or secrets enter backups.
Finish line
Verification checklist
promtool check rules {{rulesPath}}/*.yml && promtool test rules {{rulesPath}}/tests/*.test.ymlAll positive, negative, low-traffic, absent and reset fixtures pass with stable expected labels.amtool check-config {{alertmanagerConfig}}Rendered config validates without exposing receiver secrets.amtool --alertmanager.url={{alertmanagerUrl}} config routes test notification_class=test team=payments-platform severity=critical service=checkout environment=staging cluster=eu-testOnly safe-test-webhook matches and continue is false.amtool --alertmanager.url={{alertmanagerUrl}} cluster showAll three peers are ready, private and running identical configuration digests.curl --fail --silent {{alertmanagerUrl}}/api/v2/alertsSynthetic firing/resolved, grouping and inhibition outcomes reach only safe receiver; production side effects remain zero.Recovery guidance
Common problems and safe checks
A rule expression returns no series although the service is failing.
Likely causeLabel matchers, metric names or job identity changed, the target is absent, or the expression treats missing data as healthy.
Evaluate each subexpression at the same timestamp in Prometheus.Inspect target and service-discovery labels plus scrape health.
ResolutionCorrect selectors and add a separately owned absence/scrape alert where missing data is actionable; do not use `or vector(0)` without understanding label semantics.
An alert flaps between pending, firing and inactive.
Likely causeThe threshold is too close to normal variance, `for` is too short, scrape gaps reset state or the metric is not the intended signal.
Graph the expression and raw numerator/denominator across incidents and normal peaks.Inspect scrape gaps and rule evaluation duration.
ResolutionChoose a stable symptom-based signal, use an evidence-based `for`, repair scrape gaps and keep a fast page only for genuinely immediate failure.
A ratio alert spikes when traffic is low.
Likely causeThe denominator is near zero, windows differ, labels do not align or counter reset handling is wrong.
Evaluate numerator, denominator and request volume separately.Check vector matching and counter resets with `rate()` over a suitable range.
ResolutionRequire a minimum event volume, align windows/labels and unit-test zero, reset and low-traffic cases.
One incident creates hundreds of pages.
Likely causeAlert labels include instance/pod or another volatile dimension, grouping is too granular, or inhibition does not match root-cause labels.
Inspect firing alert label sets and Alertmanager group key.Preview route and inhibition matches with synthetic alerts.
ResolutionPreserve useful instance context in annotations when appropriate, group by service/cluster/alertname and add carefully scoped root-cause inhibition.
Distinct incidents are grouped into one confusing notification.
Likely causegroup_by omits cluster, environment, service or another operator decision boundary.
Compare labels of alerts inside the notification group.Use route testing to show the computed receiver and grouping labels.
ResolutionAdd stable incident dimensions to group_by without grouping by every label; test fan-out and dedup behavior.
Warning alerts continue paging during a related critical outage.
Likely causeInhibition source/target matchers or `equal` labels do not align, or a label is missing/empty on one alert.
Inspect source and target alert labels exactly, including empty values.Run a synthetic critical+warning pair against a nonproduction receiver.
ResolutionNormalize required labels in rules and use explicit source/target/equal sets; avoid broad inhibition that hides unrelated warnings.
A warning is inhibited even though no related critical incident exists.
Likely causeMissing and empty labels compare equally, matcher scope is too broad or a stale firing critical alert still matches.
Inspect `/api/v2/alerts` and exact labels of inhibitor/source.Review inhibition config and alert resolution timestamps.
ResolutionRequire stable nonempty ownership labels in both rules, narrow matchers and unit-test unrelated/missing-label cases.
Notifications repeat much more often than expected.
Likely causerepeat_interval is shorter than accepted, Alertmanager state was lost, alerts change grouping labels, or multiple independent clusters notify the same receiver.
Inspect notification log metrics, group keys and Alertmanager peer state.Compare labels and configuration revision across replicas/clusters.
ResolutionStabilize labels, align HA peer configuration/state, set deliberate repeat interval and remove duplicate independent send paths.
No reminder arrives for a long-running unresolved alert.
Likely causerepeat_interval is too long, route inherited an unexpected value, notification failed, or the alert resolved/re-fired under a new group.
Inspect effective route, notification history and receiver errors.Compare alert fingerprints and group labels over time.
ResolutionSet explicit repeat policy at the owning route and test a short nonproduction fixture before production deployment.
A test alert paged the production on-call.
Likely causeThe test label did not match the safe route, `continue` forwarded to a parent/next route, or the default receiver pages.
Disable the synthetic alert and preserve route/notification evidence.Use `amtool config routes test` with the exact synthetic labels.
ResolutionMake the top-level default a safe owned sink where feasible, put test routing first, set explicit continue behavior and require route preview before sending.
The default receiver silently discards real unmatched alerts.
Likely causeA null receiver was used as safety but unmatched production labels were not monitored.
Inspect unmatched-route metrics or a catch-all notification audit.Route-test every production severity/team combination.
ResolutionUse a monitored catch-all triage receiver, alert on unmatched ownership and keep only explicitly labeled test alerts in the null sink.
Alertmanager configuration reload fails.
Likely causeYAML syntax, receiver field, matcher syntax or secret file/reference is invalid for the pinned version.
Run `amtool check-config` against the exact rendered candidate.Inspect the reload failure metric/log without printing secret values.
ResolutionKeep the current configuration, fix and validate the candidate offline, route-test it, then reload atomically.
One Alertmanager replica shows different routes or receivers.
Likely causeConfiguration rollout is inconsistent; HA gossip shares silences/notification state, not configuration.
Compare protected configuration digest and version across every peer.Inspect cluster peer status and reload success per replica.
ResolutionDeploy the same signed configuration to every peer, validate sequentially and do not treat gossip as configuration distribution.
HA replicas send duplicate notifications during a partition.
Likely causeGossip peers cannot communicate, cluster advertise/listen addresses are wrong, or the partition exceeds dedup convergence.
Inspect cluster peer metrics and network paths on every replica.Correlate duplicate notifications with partition timestamps.
ResolutionRepair the mesh and stable peer addressing; accept/document that partitions can cause duplicates rather than hiding them with a single load-balanced replica.
Prometheus sends alerts to only one Alertmanager and loses notifications during its failure.
Likely causeService discovery/load balancing presents one selected backend instead of configuring Prometheus to send to all Alertmanager peers.
Inspect effective alerting configuration and discovered Alertmanager targets.Stop one test peer and observe send success to remaining peers.
ResolutionConfigure all HA Alertmanager peers as targets according to the supported pattern and repeat failure testing.
Notification integration returns 401 or 403.
Likely causeA webhook/token credential expired, scope changed or secret rendering produced an empty/wrong value.
Inspect receiver error metrics and provider audit metadata.Validate protected secret reference/version without printing the secret.
ResolutionRotate or restore a least-privilege credential and test through the nonproduction receiver before enabling the production route.
A secret appears in Alertmanager config backup or logs.
Likely causeThe rendered YAML embedded a token/password, verbose diagnostics printed configuration, or backups copied live secret files.
Quarantine artifacts and audit access.Scan all revisions and logs for the secret identifier without redisplaying the value.
ResolutionRotate the credential, move to protected secret files/references, regenerate secret-free artifacts and follow incident policy.
promtool test rules passes but the production alert never fires.
Likely causeFixture labels/data do not match real scrape semantics, evaluation interval differs or rule group is not loaded.
Inspect `/api/v1/rules`, last evaluation and exact expression.Capture sanitized real label shape and add it to the fixture.
ResolutionCorrect loading/labels and add a regression fixture that mirrors production identity without copying sensitive data.
A silence hides more alerts than intended.
Likely causeMatchers are broad, regex is unanchored/misunderstood, duration is excessive or the silence lacks owner/comment.
Use amtool to list the silence and matching active alerts.Review creator, comment, timestamps and exact matchers.
ResolutionExpire/delete the broad silence, create a narrowly scoped time-bounded replacement if necessary and require owner/change reference.
Alertmanager storage loss causes reminders or silences to behave unexpectedly.
Likely causeAll replicas restarted without durable state, quorum/mesh was unavailable or operators assumed HA state was a backup.
Inspect peer state, silence inventory and notification log after restart.Compare with exported operational records where policy permits.
ResolutionRestore service with consistent configuration, recreate only approved silences and document that HA replication reduces downtime but does not replace configuration/change records.
Reference
Frequently asked questions
Should every warning have a `for` duration?
Not mechanically. Choose duration from signal noise, evaluation interval and acceptable detection delay. Immediate binary failures may page without `for`; noisy ratios usually need sustained evidence and minimum volume.
Does a three-peer Alertmanager cluster guarantee one notification?
No. It normally deduplicates through shared state, but partitions can generate duplicates. HA favors delivery; operators must tolerate and monitor bounded duplication.
Why not send test alerts with `severity=critical` to production and silence them?
A silence can expire or mismatch and still page. A dedicated test label and physically nonpaging receiver provide a safer invariant.
Recovery
Rollback
Do not roll back by silencing all alerts, removing TLS/authentication or pointing production rules at a null receiver. Preserve active alert and notification evidence, restore the previous promtool/amtool-validated signed revision, and roll one evaluator/peer at a time.
- Stop the configuration rollout and preserve rule evaluation, notification queue, peer, route and receiver error evidence.
- Disable only the faulty synthetic source or rule group if it is causing load; do not apply broad silences to unrelated production alerts.
- Validate the previous rules/config with the same promtool/amtool versions and route-test the safe lane before deployment.
- Restore one Alertmanager peer at a time, verify mesh and config digest, then restore remaining peers; keep Prometheus sending to all healthy peers.
- Restore Prometheus rules to a canary, confirm inactive/pending/firing/resolved behavior with synthetic data, then stage across evaluators.
- Resume normal change flow only after safe canary delivery, production receiver health, peer membership, unmatched-route count and secret scans pass.
Evidence