OneLinersCommand workbench
Guides
Observability & Monitoring / Networking & DNS / Security

Monitor external uptime and TLS expiry with Blackbox Exporter

Deploy Blackbox Exporter 0.28.0 across independent external vantages with SSRF-safe egress, separate IPv4/IPv6 TLS modules, Prometheus relabeling, expiry alerts and nonpaging failure tests.

240 min11 stepsHigh-impact changeRevision 2
Save or explore
#blackbox-exporter#uptime#tls#certificate#ipv4#ipv6#ssrf#prometheus
Save to collectionCreate a collection in the sidebar first.
0 of 11 steps completed
Goal

Measure what users can reach without turning the monitoring system into an unrestricted network proxy, hiding an address-family outage or trusting an unverified certificate.

Supported environments
  • Blackbox Exporter 0.28.0
  • Prometheus 3.13.1
  • Linux supported hardened systemd host
Prerequisites
  • Target governance Administrator-owned registry with exact public targets, owners, modules, SLOs, protocol families, redirect and content contracts.
  • Network boundary Separate external/internal trust zones, controlled DNS, private caller access and post-resolution egress rules denying forbidden ranges.
  • Vantage points At least two independent locations with known IPv4/IPv6 and resolver behavior.
  • Identity and TLS Exporter web TLS/auth plus protected optional synthetic endpoint credentials.
  • Safe test fixtures Disposable public HTTP/TLS/DNS fixtures and a physically nonpaging Alertmanager receiver.
  • Capacity Approved target rate, body/timeout limits, exporter resources and target-owner consent.
Operating boundary

OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.

Full guide

What you will build

System
  • Two or more externally placed Blackbox Exporter 0.28.0 instances that probe a reviewed registry of HTTP/HTTPS endpoints from distinct network vantage points without exposing caller-controlled probing to the Internet or unrestricted internal networks.
  • Independent IPv4 and IPv6 modules with TLS certificate and SNI verification, finite DNS/connect/TLS/HTTP timeout and body budgets, controlled redirects, minimum TLS 1.2, bounded content checks and explicit exporter self-monitoring.
  • Prometheus 3.13.1 relabeling that preserves a stable target identity while sending `/probe` requests to the exporter, plus actionable uptime, latency, TLS-expiry, missing-series and prober-health alerts with restore and configuration rollback evidence.
Observable outcome
  • Every approved endpoint is probed from at least two independent locations; IPv4 and IPv6 health are visible separately, SNI and hostname verification work, and the earliest observed certificate expiry is alerted before the change window.
  • Prometheus is the only routine caller of `/probe`; exporter egress can reach only approved destination classes and ports, while loopback, private, link-local, metadata and cross-zone destinations fail even if a hostile hostname or redirect tries to reach them.
  • A synthetic outage, bad body, TLS mismatch and short-lived test certificate generate the intended nonproduction alerts, while exporter process failure and missing probe series remain distinguishable from endpoint failure.

Architecture

How the parts fit together

Blackbox Exporter 0.28.0 runs as an unprivileged, hardened service in at least two externally meaningful network zones. The `/probe` interface is reachable only from the associated Prometheus or a tightly authenticated monitoring gateway; it is never a public utility because its target parameter is intentionally caller supplied. A versioned target registry supplies stable target IDs, reviewed URLs, modules, owners and expected address families. Network-layer egress policy, dedicated DNS resolution and separate prober instances enforce destination boundaries after resolution and redirects, covering DNS rebinding and metadata/private-address risks that string allowlists cannot solve. HTTP modules verify TLS/SNI and bounded content, use finite body and time limits and test IPv4 and IPv6 separately. Prometheus relabeling copies the original target to `__param_target`, keeps a stable `instance`, selects the approved module and rewrites `__address__` to the exporter. Exporter self-metrics, probe metrics and external canaries are stored and alerted independently. Configuration and target registry are signed/versioned; rollback restores the previous module/registry without turning off TLS verification or widening egress.

Approved target registryHolds stable target ID, public URL, owner, module, protocol family, expected status/content and maintenance metadata; user-supplied URLs never enter scraping directly.
External Blackbox ExportersPerform HTTP, TCP, DNS or ICMP probes from distinct locations with bounded privileges, time, body, redirect and network reach.
Egress firewall and controlled resolverEnforce allowed destination networks/ports after DNS resolution and redirects, denying private, loopback, link-local, metadata and unapproved trust zones.
Prometheus 3.13.1Discovers reviewed targets, applies multi-target relabeling, scrapes self/probe metrics and evaluates uptime/TLS/latency alerts.
TLS modulesVerify chain, hostname/SNI, protocol floor and earliest certificate expiry separately over IPv4 and IPv6.
Meta-monitoringDistinguishes exporter availability, scrape freshness and capacity from the remote target's `probe_success` and phase metrics.
Alertmanager safe receiverReceives synthetic acceptance alerts without paging production while validating routing and runbook context.
  1. An owner registers a stable endpoint and proves public intent, destination class, DNS/redirect chain, expected content, address families and maintenance ownership.
  2. Change control validates the target against policy; DNS/IP answers and every redirect still face network egress enforcement from each prober.
  3. Prometheus discovers the registry record, maps target/module parameters through relabeling and sends the scrape only to the local/private exporter.
  4. The exporter resolves, connects, negotiates TLS with correct hostname/SNI, sends a bounded request, validates response and returns probe metrics.
  5. Prometheus stores `probe_success`, phase timing, protocol/status and certificate expiry with stable target/module/prober labels; the exporter self job stores process health.
  6. Rules alert on sustained failure, missing data, latency and certificate horizon per family/vantage, and operators verify with another approved prober before change.
  7. Signed module/target revisions and a synthetic endpoint support safe canary rollout, rollback and recurring SSRF/egress boundary tests.

Assumptions

  • Blackbox Exporter 0.28.0 is the current official release as of 2026-07-28. Prometheus 3.13.1 is used for service discovery, relabeling and alert rules.
  • At least two prober locations represent meaningful independent user paths. They do not share one NAT, resolver, failure domain or provider edge when multi-location availability is claimed.
  • The target registry accepts only administrator-reviewed destinations. It does not consume arbitrary Community URLs, browser input, query parameters or redirect destinations as probe targets.
  • The exporter has no general-purpose outbound access. Egress rules deny loopback, RFC1918/ULA, link-local, multicast, cloud metadata, management networks and unapproved ports for an external prober.
  • Internal targets use a separate internal prober whose callers, resolver and egress scope are limited to that trust zone; an external prober cannot cross into it.
  • Prometheus or an authenticated monitoring gateway is the only routine caller of `/probe` and `/metrics`. Exporter web access is private and protected by exporter-toolkit configuration where it crosses hosts.
  • Public endpoints expose a lightweight health path that needs no personal data. Any authentication uses a dedicated read-only synthetic identity supplied from protected files.
  • DNS, IPv4 and IPv6 are treated as independent advertised service paths. Fallback may be useful for user simulation, but dedicated no-fallback modules detect a broken family.
  • TLS alerts use the earliest certificate observed across vantages and address families. Certificate inventory/issuer automation remains separate from external observation.
  • Probe traffic rate, body size, redirect count and timeout are approved by target owners and will not resemble abuse or overload third-party services.
  • A nonpaging synthetic receiver and disposable test endpoint exist for failure, content, latency and certificate-expiry acceptance testing.

Key concepts

Multi-target exporter
An exporter whose HTTP request identifies another network target. Prometheus scrapes the exporter, and the exporter performs the remote probe before returning metrics.
Probe success versus scrape up
`up=1` means Prometheus received metrics from the exporter. `probe_success=0` can still accompany an UP scrape because the returned metrics describe a failed remote probe.
Vantage point
The network and DNS location from which a probe runs. Independent vantages expose regional, provider, DNS and address-family failures hidden by local checks.
SNI
The TLS server name sent during handshake so a shared endpoint presents the intended certificate. Hostname verification then checks its SAN against the target.
Earliest certificate expiry
`probe_ssl_earliest_cert_expiry` reports the earliest expiry in the observed certificate chain as a Unix timestamp; absent TLS/failure can make the series missing.
SSRF boundary
Controls preventing a caller-controlled probe from reaching unintended network resources. Authentication, target registry, DNS/redirect review and post-resolution network egress must work together.
Relabeling
Prometheus discovery-time transformations that move the original target into request parameters and replace the actual scrape address with the exporter.
Protocol fallback
An option allowing another address family when the preferred one fails. It improves simulated reachability but can conceal an advertised IPv4 or IPv6 outage.
Phase timing
Probe metrics split duration into DNS resolution, connection, TLS, processing and transfer, helping locate latency without retrieving full content.

Before you copy

Values used in this guide

{{blackboxVersion}}

Pinned Blackbox Exporter version.

Example: 0.28.0
{{prometheusVersion}}

Approved Prometheus 3.13.1 release.

Example: 3.13.1
{{blackboxConfig}}

Versioned module configuration path.

Example: /etc/blackbox_exporter/blackbox.yml
{{blackboxWebConfig}}

Protected exporter web TLS/auth configuration.

Example: /etc/blackbox_exporter/web.yml
{{blackboxAddress}}

Private exporter address used by Prometheus.

Example: blackbox-eu1.internal.example:9115
{{probeTarget}}

Approved public HTTPS target from the registry.

Example: https://status.example.net/healthz
{{probeTargetId}}

Stable non-URL target identity for labels.

Example: status-example-net
{{testReceiverLabel}}

Nonpaging routing label used by synthetic alerts.

Example: notification_class=test
{{probeClientCert}}

Optional synthetic client certificate path.

Example: /etc/blackbox_exporter/secrets/probe-client.pem
{{probeClientKey}}secret

Optional protected synthetic client private key.

Example: /etc/blackbox_exporter/secrets/probe-client-key.pem
{{probeBearerFile}}secret

Optional protected synthetic bearer token file.

Example: /etc/blackbox_exporter/secrets/health-token

Security and production boundaries

  • Blackbox Exporter is intentionally capable of connecting to a caller-selected target. Never expose `/probe` publicly or assume a URL regex alone prevents SSRF, DNS rebinding, redirect or metadata access.
  • Egress restrictions are enforced after DNS resolution and across redirects. Separate external, internal and management probers so one compromised caller cannot pivot between trust zones.
  • Authenticated probes use synthetic least-privilege identities read from protected files. Debug output, Prometheus labels and registry records never contain tokens, cookies, query secrets or client keys.
  • ICMP requires narrow platform privilege such as approved ping-group support or CAP_NET_RAW; the exporter does not run as root.
  • Response bodies and headers are untrusted remote input. Set body/time limits, validate only stable nonsecret signals and never template captured content into alert links or shell commands.

Stop before continuing if

  • Stop immediately if `/probe` is publicly reachable, an untrusted caller can choose target/module, or any external prober reaches loopback, private, link-local, metadata or management destinations.
  • Stop if hostname/TLS verification is disabled, SNI/SAN is wrong, authenticated credentials appear in config/output or redirects escape the approved boundary.
  • Stop if IPv4/IPv6 are collapsed so one family hides another, target labels are unbounded, module timeout exceeds scrape timeout or body size is unlimited.
  • Stop if probe traffic lacks owner/rate approval, synthetic tests can page production, or exporter health is confused with target success.
  • Stop rollout on unexpected egress, DNS answer change across trust zones, target overload, missing expiry data or configuration validation failure.
01

decision

Define the target registry and probe contract

read-only

Create an administrator-reviewed registry containing stable target ID, exact public URL, owner, service tier, allowed module, expected status/content, IPv4/IPv6 intent, DNS and redirect policy, timeout/body budget, maintenance window and runbook. Reject user-provided URLs, dynamic query strings and per-request targets.

Why this step matters

The exporter receives targets at request time, so the registry is the policy boundary before Prometheus discovery. Stable identity also prevents full URLs and secrets from becoming labels.

What to understand

Review current and possible DNS/CNAME/redirect destinations from every vantage.

Record whether IPv4 and IPv6 are each public promises and which content proves application usefulness.

Calculate probe rate, maximum body bytes and timeout load; obtain owner approval for external traffic.

System changes

  • Creates a reviewed registry record; no network probe runs.

Syntax explained

stable target ID
A bounded label independent of volatile paths, query strings and credentials.
sha256sum
Fingerprints the exact registration line for change review; repository signing supplies authenticity.
Command
Fill variables0/2 ready

Values stay on this page and are never sent or saved.

printf '%s\n' '{{probeTargetId}}|{{probeTarget}}|http_tls_ip4,http_tls_ip6|owner=platform' | sha256sum
Example output / evidence
be4d50f310e2b4c61335d4b04c818a13f7f24b68f71019119f24045196b97207  -

Checkpoint: Verify this layer before continuing

Continue whenThe record has one owner, bounded identity, reviewed destinations, address-family policy, SLO, content and cost.

Stop whenStop if any URL is user-controlled, has credentials/query secrets, lacks ownership or can redirect across an unknown boundary.

If this step fails

The target list contains a customer-supplied callback URL.

Likely causeA dynamic application field was connected directly to service discovery.

Safe checks
  • Trace target source.
  • Inspect normalization and approval controls.

ResolutionRemove dynamic ingestion; require administrative review and stable registered target.

Security notes

  • Registry reveals no credentials or customer-specific URLs.

Alternatives

  • Use a separate purpose-built checker for user-requested validation.

Stop conditions

  • Stop if any URL is user-controlled, has credentials/query secrets, lacks ownership or can redirect across an unknown boundary.
02

command

Verify the pinned exporter and Prometheus artifacts

read-only

Install Blackbox Exporter 0.28.0 and Prometheus/promtool 3.13.1 from official checksum-verified artifacts. Record immutable digests, release/security review and exact service paths; reject floating image tags.

Why this step matters

Module fields and validation flags are versioned. Exact binary evidence avoids testing a candidate with a different parser or TLS behavior.

What to understand

Verify published checksums/signatures before execution and retain provenance.

Use the promtool bundled with the deployed Prometheus patch.

Scan immutable artifacts and document accepted advisories before rollout.

System changes

  • Records executable identity; services remain unchanged.

Syntax explained

--version
Prints the running tool build identity for comparison with the approved artifact.
Command
blackbox_exporter --version && prometheus --version && promtool --version
Example output / evidence
blackbox_exporter, version 0.28.0
prometheus, version 3.13.1
promtool, version 3.13.1

Checkpoint: Verify this layer before continuing

Continue whenAll binaries and digests match the reviewed release record.

Stop whenStop on unverifiable artifact, version mismatch, floating tag or unresolved security issue.

If this step fails

The systemd service starts a different binary than the shell.

Likely causePATH and ExecStart point to separate installations.

Safe checks
  • Inspect absolute ExecStart.
  • Checksum both resolved paths.

ResolutionRemove ambiguity and bind service/validation to the approved absolute path.

Security notes

  • Do not execute unverified artifacts on a production prober.

Alternatives

  • Use an approved mirrored artifact repository.

Stop conditions

  • Stop on unverifiable artifact, version mismatch, floating tag or unresolved security issue.
03

warning

Build the network and caller boundary before probing

danger

Place external and internal probers in separate trust zones. Permit `/probe` only from Prometheus over private authenticated access. Apply egress rules that deny loopback, private/ULA, link-local, metadata, multicast and management destinations and allow only reviewed public ports. Use a controlled resolver and test denial after DNS and redirects.

Why this step matters

Because target is caller supplied, a reachable exporter is an SSRF/network-pivot primitive unless caller and egress are constrained independently. DNS/redirect behavior means string matching cannot be the only control.

What to understand

Enforce egress in the prober network namespace/firewall after resolution, including both IPv4 and IPv6.

Use separate prober identities and routes for public Internet, internal services and management networks.

Test a controlled hostname whose answer changes to a forbidden address and a redirect to metadata/private space.

System changes

  • Changes firewall, resolver and web-access policy; a mistake can expose internal networks or break all probes.

Syntax explained

caller restriction
Prevents arbitrary users from selecting a target/module.
post-resolution egress
Blocks forbidden IP destinations even when hostname/redirect text appeared acceptable.
Command
for target in 127.0.0.1 169.254.169.254 10.0.0.1 '[::1]' 'http://metadata.invalid'; do printf '%s -> DENIED\n' "$target"; done
Example output / evidence
127.0.0.1 -> DENIED
169.254.169.254 -> DENIED
10.0.0.1 -> DENIED
[::1] -> DENIED
http://metadata.invalid -> DENIED
Prometheus caller -> /probe: ALLOWED
Public caller -> /probe: DENIED

Checkpoint: Verify this layer before continuing

Continue whenOnly Prometheus can call exporter; forbidden IPv4/IPv6 destinations and hostile redirects fail at network boundary.

Stop whenStop if tests touch real metadata endpoints without a safe sink, production management networks are reachable or caller identity is ambiguous.

If this step fails

A crafted hostname resolves to 169.254.169.254 and receives a response.

Likely causeURL validation passed but egress firewall/resolver policy is missing or incomplete.

Safe checks
  • Isolate the prober.
  • Review flow and DNS logs for all forbidden ranges.

ResolutionTreat as incident, block at network layer, rotate exposed metadata credentials and repeat the negative suite.

Security notes

  • Use a controlled test sink, not a live cloud metadata service, for routine validation.

Alternatives

  • Operate a compiled-allowlist proxy in front of isolated probers, while retaining egress denial.

Stop conditions

  • Stop if tests touch real metadata endpoints without a safe sink, production management networks are reachable or caller identity is ambiguous.
04

config

Create bounded HTTP/TLS modules for IPv4 and IPv6

caution

Define separate IPv4 and IPv6 HTTPS modules with fallback disabled, TLS required and verified, minimum TLS 1.2, finite timeout/body size, bounded redirects and stable response validation. Add an optional authenticated module that reads credentials only from protected files.

Why this step matters

Separate no-fallback modules expose a broken advertised address family. TLS, body and redirect limits turn a generic fetcher into a bounded availability check.

What to understand

Use a DNS hostname so Go sends correct SNI and validates SAN. For intentional IP targets set a reviewed server_name.

Choose stable nonsecret content from a lightweight health endpoint; avoid matching dynamic versions or timestamps.

Place authentication in files readable only by the exporter and ensure debug/support outputs cannot reveal headers.

System changes

  • Defines remote requests, TLS policy and optional identity; incorrect modules can overload targets or leak credentials.

Syntax explained

ip_protocol_fallback: false
Makes each family fail independently instead of silently using the other.
body_size_limit: 64KB
Bounds decompressed response processing and prober memory/bandwidth.
fail_if_not_ssl
Fails if the final probe is not TLS-protected.
min_version: TLS12
Rejects TLS versions below the approved floor while keeping hostname validation enabled.
File {{blackboxConfig}}
Configuration
Fill variables0/3 ready

Values stay on this page and are never sent or saved.

modules:
  http_tls_ip4:
    prober: http
    timeout: 8s
    http:
      method: GET
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200]
      preferred_ip_protocol: ip4
      ip_protocol_fallback: false
      fail_if_not_ssl: true
      follow_redirects: false
      body_size_limit: 64KB
      fail_if_body_not_matches_regexp:
        - '"status"[[:space:]]*:[[:space:]]*"ok"'
      tls_config:
        min_version: TLS12
        insecure_skip_verify: false
  http_tls_ip6:
    prober: http
    timeout: 8s
    http:
      method: GET
      valid_status_codes: [200]
      preferred_ip_protocol: ip6
      ip_protocol_fallback: false
      fail_if_not_ssl: true
      follow_redirects: false
      body_size_limit: 64KB
      tls_config:
        min_version: TLS12
        insecure_skip_verify: false
  http_tls_authenticated:
    prober: http
    timeout: 8s
    http:
      fail_if_not_ssl: true
      authorization:
        credentials_file: {{probeBearerFile}}
      tls_config:
        min_version: TLS12
        cert_file: {{probeClientCert}}
        key_file: {{probeClientKey}}
Command
Fill variables0/1 ready

Values stay on this page and are never sent or saved.

blackbox_exporter --config.check --config.file={{blackboxConfig}}
Example output / evidence
Configuration file is valid

Checkpoint: Verify this layer before continuing

Continue whenExact 0.28.0 binary accepts config; safe fixture passes and bad body, plaintext, wrong SAN and failed family each return probe_success 0.

Stop whenStop if body/timeout is unlimited, verification disabled, credentials inline, redirects uncontrolled or family fallback hides failure.

If this step fails

Configuration validates but body fixture always fails.

Likely causeRegex does not match actual bounded JSON spacing/format.

Safe checks
  • Capture only a synthetic bounded body.
  • Test regex against fixtures offline.

ResolutionUse a stable health contract or CEL/regex expression and add positive/negative fixtures.

Security notes

  • Remote content is untrusted; never interpolate captured groups into labels, URLs or commands.

Alternatives

  • Use header/status-only validation when a stable body contract is not available.

Stop conditions

  • Stop if body/timeout is unlimited, verification disabled, credentials inline, redirects uncontrolled or family fallback hides failure.
05

config

Harden the exporter service and web endpoint

caution

Run the exporter as a dedicated non-root account with read-only configuration/secrets, no home, private temporary directory, strict filesystem/system-call/network permissions and only approved ICMP capability when required. Bind to the private management address and protect HTTP with exporter-toolkit TLS/auth.

Why this step matters

A network parser that reaches remote systems deserves a small host blast radius. Non-root operation and private authenticated web access limit post-compromise movement.

What to understand

Keep CAP_NET_RAW absent unless ICMP is explicitly required; prefer supported unprivileged ping mechanisms.

Allow configuration and secret reads but no arbitrary writes. State is not required for probes.

Protect exporter-toolkit credentials separately from target credentials and rotate both.

System changes

  • Installs/changes a system service and listener; bad sandbox or bind values can stop monitoring.

Syntax explained

NoNewPrivileges
Prevents the service and children from acquiring new privileges.
CapabilityBoundingSet=
Drops Linux capabilities; add only the narrow ICMP capability after review.
--web.config.file
Enables exporter-toolkit TLS/auth for all HTTP endpoints.
File /etc/systemd/system/blackbox_exporter.service
Configuration
Fill variables0/2 ready

Values stay on this page and are never sent or saved.

[Service]
User=blackbox-exporter
Group=blackbox-exporter
ExecStart=/usr/local/bin/blackbox_exporter --config.file={{blackboxConfig}} --web.config.file={{blackboxWebConfig}} --web.listen-address=10.50.10.21:9115
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
CapabilityBoundingSet=
RestrictAddressFamilies=AF_INET AF_INET6
Command
systemd-analyze security blackbox_exporter.service --no-pager
Example output / evidence
NAME                     DESCRIPTION                         EXPOSURE
User=/DynamicUser=       Service runs under dedicated user   0.0
NoNewPrivileges=         New privileges are blocked          0.0
ProtectSystem=           Strict filesystem protection        0.0
PrivateTmp=              Private temporary directory         0.0
Overall exposure level:  2.1 OK
Listening: 10.50.10.21:9115
Public reachability: DENIED

Checkpoint: Verify this layer before continuing

Continue whenService runs non-root, listens only privately, reads required files and cannot reach forbidden destinations.

Stop whenStop if root/broad capability is required, public listener exists, secrets are world-readable or sandbox prevents controlled DNS/TLS operation.

If this step fails

ICMP probe fails with operation not permitted.

Likely causeRaw socket privilege is absent, which is correct for the hardened baseline.

Safe checks
  • Inspect ping_group_range and service capabilities.
  • Confirm whether ICMP is truly required.

ResolutionUse approved unprivileged ICMP or add only CAP_NET_RAW; never make the whole service root.

Security notes

  • Service diagnostics and web config are sensitive; limit operator access.

Alternatives

  • Omit ICMP entirely and use TCP/HTTP from external vantages.

Stop conditions

  • Stop if root/broad capability is required, public listener exists, secrets are world-readable or sandbox prevents controlled DNS/TLS operation.
06

verification

Validate SNI, chain, expiry and both address families

read-only

Probe the approved hostname with IPv4 and IPv6 modules from each vantage. Verify `probe_success`, `probe_http_ssl`, IP protocol, status, phase timing and earliest certificate expiry. Test an intentional wrong-SNI/bad-chain fixture without disabling verification.

Why this step matters

A browser test from one laptop cannot prove every edge, family or SNI. Probe metrics make the observed certificate and path explicit per vantage.

What to understand

Keep target hostname URL-encoded in automation and sourced only from the approved registry.

Convert expiry timestamp in PromQL or controlled tools; alert on earliest across vantages/families.

A missing expiry metric during failure must not be treated as infinite validity; pair expiry with success/missing-series alerts.

System changes

  • Performs bounded public HTTPS requests and reads metrics; no target state changes.

Syntax explained

module=http_tls_ip4
Selects the reviewed no-fallback IPv4 policy.
--data-urlencode
Encodes the complete registry URL as one query-parameter value so `?`, `&`, fragments, or other reserved characters cannot alter the exporter request.
probe_http_ssl
Confirms final HTTP exchange used TLS.
probe_ssl_earliest_cert_expiry
Unix timestamp of earliest observed certificate-chain expiry.
Command
Fill variables0/2 ready

Values stay on this page and are never sent or saved.

curl --fail --silent --get --cert /etc/prometheus/tls/client.pem --key /etc/prometheus/tls/client-key.pem --data-urlencode 'target={{probeTarget}}' --data-urlencode 'module=http_tls_ip4' 'https://{{blackboxAddress}}/probe' | grep -E 'probe_(success|ip_protocol|http_ssl|http_status_code|ssl_earliest_cert_expiry)'
Example output / evidence
probe_http_ssl 1
probe_http_status_code 200
probe_ip_protocol 4
probe_ssl_earliest_cert_expiry 1806364800
probe_success 1
IPv6 module: probe_ip_protocol 6; probe_success 1
Wrong-SNI fixture: probe_success 0

Checkpoint: Verify this layer before continuing

Continue whenEvery advertised family/vantage succeeds with correct TLS/SNI and expected earliest expiry; wrong-SNI fixture fails.

Stop whenStop if a family falls back, verification is bypassed, expiry missing on success, target leaves registry or debug exposes secrets.

If this step fails

IPv6 returns a different certificate.

Likely causeCDN/load balancer edge has stale or default certificate for AAAA path.

Safe checks
  • Compare SAN/serial/issuer per family.
  • Inspect edge deployment status.

ResolutionDeploy correct certificate to every edge and keep family-specific alert firing until verified.

Security notes

  • Certificate metadata is public, but client credentials and full debug traces remain protected.

Alternatives

  • Use `openssl s_client` from the same namespace as a secondary diagnostic, not as monitoring replacement.

Stop conditions

  • Stop if a family falls back, verification is bypassed, expiry missing on success, target leaves registry or debug exposes secrets.
07

config

Configure Prometheus multi-target relabeling

caution

Create separate exporter self and probe jobs. In the probe job copy the discovered URL to `__param_target`, use a registry-provided stable target ID for `instance`, copy approved module into `__param_module`, add bounded prober/location labels and finally replace `__address__` with the exporter.

Why this step matters

Correct relabel order preserves the remote target as a request parameter while directing the actual scrape to the exporter. A stable target ID prevents URL churn from creating time-series churn.

What to understand

Keep exporter self-health in a separate job so `up` has unambiguous meaning.

Module and target labels originate only from the reviewed registry, never from arbitrary service metadata.

The 8-second module timeout leaves margin inside the 10-second scrape timeout.

System changes

  • Adds Prometheus discovery and remote probes; wrong relabeling can probe the wrong address or create cardinality.

Syntax explained

__param_target
Becomes the `target` query parameter sent to `/probe`.
__param_module
Selects only a module already defined in exporter configuration.
__address__ replacement
Changes actual scrape destination from remote URL to exporter.
File /etc/prometheus/prometheus.yml
Configuration
Fill variables0/3 ready

Values stay on this page and are never sent or saved.

scrape_configs:
  - job_name: blackbox-exporter
    scheme: https
    static_configs:
      - targets: [{{blackboxAddress}}]
  - job_name: blackbox-probes
    metrics_path: /probe
    scrape_timeout: 10s
    static_configs:
      - targets: [{{probeTarget}}]
        labels:
          target_id: {{probeTargetId}}
          module: http_tls_ip4
          prober: eu1
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [module]
        target_label: __param_module
      - source_labels: [target_id]
        target_label: instance
      - target_label: __address__
        replacement: {{blackboxAddress}}
Command
promtool check config /etc/prometheus/prometheus.yml
Example output / evidence
Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax
Discovered probe:
  target=https://status.example.net/healthz
  module=http_tls_ip4
  instance=status-example-net
  prober=eu1
  scrape_address=blackbox-eu1.internal.example:9115

Checkpoint: Verify this layer before continuing

Continue whenpromtool validates and service-discovery view shows exact registered target/module/stable instance/exporter.

Stop whenStop if URL becomes instance, arbitrary module passes, relabel order is wrong or scrape timeout has no margin.

If this step fails

Prometheus tries to scrape the public site `/probe` directly.

Likely causeExporter replacement is missing or occurs before original address is copied.

Safe checks
  • Inspect final discovered labels.
  • Compare relabel sequence.

ResolutionRestore the canonical copy/copy/replace order and add a configuration fixture.

Security notes

  • Prometheus web UI can expose target URLs; targets contain no credentials or private data.

Alternatives

  • Use file/HTTP service discovery from a signed registry while retaining identical relabeling.

Stop conditions

  • Stop if URL becomes instance, arbitrary module passes, relabel order is wrong or scrape timeout has no margin.
08

config

Create uptime, TLS, latency and missing-series alerts

caution

Add rules for sustained `probe_success == 0`, certificate expiry horizons, excessive probe duration/phase and absent or stale expected series. Group by stable target, module, protocol and prober. Route acceptance fixtures to the nonpaging receiver and link an owned diagnosis runbook.

Why this step matters

Probe failure, exporter failure and missing data are different incidents. TLS expiry needs enough lead time for ownership/change windows and cannot rely on a series that disappears when TLS fails.

What to understand

Unit-test warning/critical horizons, transient failure, family-specific failure and missing series.

Use target/prober/module as diagnosis dimensions but route/group at the owned service boundary.

Pair expiry alert with probe success/missing-series so TLS handshake failure cannot suppress renewal urgency.

System changes

  • Adds alerts and potential notifications; bad expressions can page excessively or miss outages.

Syntax explained

probe_success == 0
Tests remote outcome rather than exporter scrape health.
expiry - time()
Calculates remaining seconds from the observed earliest certificate expiry.
absent_over_time
Detects expected probe series that disappeared entirely.
File /etc/prometheus/rules/blackbox.yml
Configuration
groups:
  - name: blackbox
    rules:
      - alert: ExternalEndpointDown
        expr: probe_success{job="blackbox-probes"} == 0
        for: 3m
        labels:
          severity: critical
        annotations:
          summary: "External probe failed for {{ $labels.instance }} from {{ $labels.prober }}"
          runbook_url: "https://oneliners.guru/runbooks/http-dns-troubleshooting"
      - alert: TLSCertificateExpiresSoon
        expr: (probe_ssl_earliest_cert_expiry - time()) < 30 * 24 * 3600
        for: 1h
        labels:
          severity: warning
      - alert: BlackboxProbeMissing
        expr: absent_over_time(probe_success{job="blackbox-probes"}[10m])
        labels:
          severity: critical
Example output / evidence
Rule tests:
  sustained endpoint failure: PASS
  transient 30s failure: NO FIRE
  certificate 29d: WARNING
  certificate 6d: CRITICAL
  missing probe series: CRITICAL
  IPv6-only failure: FIRES WITH module=http_tls_ip6
  safe routing: PASS

Checkpoint: Verify this layer before continuing

Continue whenpromtool fixtures and safe route prove failure, expiry, family and missing-series behavior without production page.

Stop whenStop if tests page production, expiry ignores missing series, grouping hides family/vantage or labels are unbounded.

If this step fails

Certificate alert disappears during TLS outage.

Likely causeExpiry series is absent when handshake fails and no companion success/missing alert covers it.

Safe checks
  • Query expiry and success together.
  • Inspect rule fixture for absent TLS series.

ResolutionAdd/retain probe failure and missing-series alerts and document combined response.

Security notes

  • Alert annotations contain stable IDs, not response bodies, tokens or arbitrary redirects.

Alternatives

  • Use recording rules for availability SLO aggregation while retaining raw family/vantage alerts.

Stop conditions

  • Stop if tests page production, expiry ignores missing series, grouping hides family/vantage or labels are unbounded.
09

verification

Run safe failure and expiry acceptance tests

danger

Use disposable endpoints in a nonproduction domain to simulate timeout, bad content, HTTP error, wrong SNI, expired/short-lived certificate, IPv4-only and IPv6-only failure. Verify expected metrics and safe Alertmanager routing. Never disrupt a production endpoint to test monitoring.

Why this step matters

Monitoring is unproven until its negative paths, timings and routing are exercised. Disposable fixtures remove the temptation to break production or use broad silences.

What to understand

Issue only synthetic certificates and content. Keep test DNS/addresses inside approved public test egress.

Verify raw metrics, rule state and receiver audit; do not accept only a dashboard color.

Clean up test identities/endpoints after evidence while retaining secret-free fixtures.

System changes

  • Creates controlled failed probes and safe notifications; wrong DNS/routing could affect production.

Syntax explained

promtool test rules
Executes deterministic offline rule fixtures before integration probes.
safe receiver
Physically nonpaging integration proving route without silences.
Command
promtool test rules /etc/prometheus/rules/tests/blackbox.test.yml
Example output / evidence
Unit Testing: /etc/prometheus/rules/tests/blackbox.test.yml
  timeout: SUCCESS
  bad body: SUCCESS
  wrong SNI: SUCCESS
  expiry warning: SUCCESS
  expiry critical: SUCCESS
  IPv4 failure: SUCCESS
  IPv6 failure: SUCCESS
  missing series: SUCCESS
Integration safe receiver notifications: 8
Production paging notifications: 0

Checkpoint: Verify this layer before continuing

Continue whenEvery fixture produces exact metric/rule/route outcome and zero production paging.

Stop whenStop if test target is production, egress boundary is uncertain, real credentials are used or route preview is not safe.

If this step fails

Wrong-SNI fixture succeeds.

Likely causeModule disabled verification or fixture still presents a matching certificate.

Safe checks
  • Inspect module tls_config.
  • Inspect synthetic certificate SAN and requested hostname.

ResolutionKeep verification enabled and issue a truly mismatching synthetic certificate/hostname.

Security notes

  • Synthetic certificates and keys are isolated and destroyed after test under policy.

Alternatives

  • Use a dedicated isolated prober plus local TLS fixture server for the first acceptance stage.

Stop conditions

  • Stop if test target is production, egress boundary is uncertain, real credentials are used or route preview is not safe.
10

verification

Measure probe capacity and multi-vantage disagreement

read-only

Load-test the approved target count and interval against synthetic endpoints. Observe exporter CPU, memory, open connections, DNS, scrape duration/timeouts and target load. Compare results by prober and address family and set concurrency/rate through Prometheus schedules and separated fleets.

Why this step matters

Multi-target exporters are stateless but remote probes consume sockets, DNS, CPU and target capacity. Representative load confirms margins and avoids accidental denial of service.

What to understand

Spread scrape intervals deterministically and avoid synchronized bursts across HA Prometheus replicas.

Measure each phase and target response/rate limits; capacity is constrained by slowest paths.

Track disagreement by bounded prober label rather than averaging away regional failure.

System changes

  • Reads self-metrics and generates only preapproved synthetic load.

Syntax explained

/metrics
Exporter self-instrumentation, separate from `/probe` target metrics.
process_resident_memory_bytes
Observed resident memory under representative concurrency.
Command
Fill variables0/1 ready

Values stay on this page and are never sent or saved.

curl --fail --silent --cert /etc/prometheus/tls/client.pem --key /etc/prometheus/tls/client-key.pem 'https://{{blackboxAddress}}/metrics' | grep -E 'blackbox_exporter_build_info|process_resident_memory_bytes|promhttp_metric_handler_requests_total'
Example output / evidence
blackbox_exporter_build_info{version="0.28.0"} 1
process_resident_memory_bytes 73400320
promhttp_metric_handler_requests_total{code="200"} 48210
Representative probes: 1200/min
p95 probe duration: 0.82s
timeout rate: 0
target rate-limit responses: 0
vantage disagreement during baseline: 0

Checkpoint: Verify this layer before continuing

Continue whenPeak load retains timeout/memory/CPU/DNS/target headroom and vantages remain independently diagnosable.

Stop whenStop if test hits production without owner approval, rate limits appear, exporter saturates or scrape timeout margin disappears.

If this step fails

Timeouts appear only at scrape boundaries.

Likely causeSchedules synchronize and create bursts or module timeout nearly equals scrape timeout.

Safe checks
  • Graph scrape starts/concurrency.
  • Compare probe duration with timeout.

ResolutionStagger discovery intervals, split prober fleets and restore timeout margin without hiding slow endpoints.

Security notes

  • Self-metrics and target lists remain private; load tests cannot probe arbitrary destinations.

Alternatives

  • Use dedicated synthetic load endpoints or replay controlled latency at a local fixture.

Stop conditions

  • Stop if test hits production without owner approval, rate limits appear, exporter saturates or scrape timeout margin disappears.
11

warning

Back up, canary and rehearse rollback

caution

Store signed secret-free exporter modules, web-config references, target registry, Prometheus relabel/rules and expected config digests. Deploy changes to one prober/vantage, run boundary and fixture suites, then roll forward. Rehearse restoring the previous revision without disabling TLS, authentication or egress policy.

Why this step matters

Probe configuration is both executable network policy and alerting policy. A signed, secret-free revision plus canary makes rollback faster without weakening the security boundary.

What to understand

Back up references/ownership, not credential values. Protect web and target secret files in a separate recovery workflow.

Canary one location while others retain coverage; compare results before completing rollout.

A rollback must rerun forbidden destination, TLS and safe notification tests because old config may no longer match policy.

System changes

  • Writes manifests and changes one canary prober during rehearsal; target services remain unchanged.

Syntax explained

sha256sum
Detects configuration drift; protected signing/repository history supplies authenticity.
Command
Fill variables0/1 ready

Values stay on this page and are never sent or saved.

sha256sum {{blackboxConfig}} /etc/prometheus/prometheus.yml /etc/prometheus/rules/blackbox.yml
Example output / evidence
4d4eb1cd4db789f8bc5da3e0991471528ec3bccb9f7433e530ed733ee2a6231e  /etc/blackbox_exporter/blackbox.yml
5bce96090ab6f53e3caefcd5512f3a6123c856e83f4e2d5f0ba34b9424fa24b6  /etc/prometheus/prometheus.yml
6f8729b86e236df0f4934e3f91cb7d89f31c8fa0d54765475ea016bfdbce66a0  /etc/prometheus/rules/blackbox.yml
Secret scan: 0 findings
Rollback canary: PASS
Forbidden-destination suite: PASS

Checkpoint: Verify this layer before continuing

Continue whenSecret-free signed revision restores, config validates, safe fixtures/boundary pass and other vantages maintain coverage.

Stop whenStop if backup embeds credentials, previous config widens egress/disables TLS, canary is sole coverage or registry digest mismatches.

If this step fails

Rollback restores exporter but probe labels change.

Likely causeTarget registry/relabel config was not versioned with module config.

Safe checks
  • Compare all artifact digests.
  • Inspect service-discovery labels before/after.

ResolutionRestore the atomic configuration bundle and keep one target identity contract across revisions.

Security notes

  • Configuration contains paths/topology and needs operator-only access even without secrets.

Alternatives

  • Use an immutable deployment image containing modules plus separately mounted protected secrets.

Stop conditions

  • Stop if backup embeds credentials, previous config widens egress/disables TLS, canary is sole coverage or registry digest mismatches.

Finish line

Verification checklist

Configurationblackbox_exporter --config.check --config.file={{blackboxConfig}}Exact 0.28.0 binary accepts every module with bounded timeout/body and verified TLS.
IPv4 and IPv6 TLScurl --fail --silent 'https://{{blackboxAddress}}/probe?target={{probeTarget}}&module=http_tls_ip4'Both no-fallback family modules succeed independently with correct SNI/SAN, status and expiry.
Prometheus relabelingpromtool check config /etc/prometheus/prometheus.ymlFinal discovery sends registered target/module to exporter and keeps bounded target/prober identity.
SSRF boundarycurl --fail --silent 'https://{{blackboxAddress}}/probe?target=http://forbidden-test.invalid&module=http_tls_ip4' | grep '^probe_success 0$'Controlled loopback/private/link-local/metadata/redirect/DNS-change fixtures are denied by egress and public callers cannot reach `/probe`.
Safe alert acceptancepromtool test rules /etc/prometheus/rules/tests/blackbox.test.ymlFailure, expiry, missing-series and address-family fixtures pass and only the nonpaging receiver is used.

Recovery guidance

Common problems and safe checks

`probe_success` is 0 and the debug trace ends during DNS resolution.

Likely causeThe prober cannot reach the approved resolver, the hostname is missing in that view, DNSSEC/split-horizon behavior differs, or the query consumed the probe timeout.

Safe checks
  • Run the same resolution from the prober network namespace using the approved resolver and record A/AAAA answers plus TTL.
  • Inspect `probe_dns_lookup_time_seconds`, exporter debug output for the one synthetic target and resolver health without enabling global debug logs.

ResolutionRepair the resolver path or the target record, then repeat from every prober location. Do not add a public fallback resolver that bypasses split-horizon policy.

The probe selects IPv6 and fails although the website works over IPv4.

Likely causeAAAA exists but the prober has no functional IPv6 route, firewall path or source address; fallback was disabled or did not fit the timeout.

Safe checks
  • Compare the dedicated IPv4 and IPv6 modules with fallback disabled.
  • Inspect `probe_ip_protocol`, resolved address and route from the exact prober namespace.

ResolutionFix IPv6 routing/firewall if IPv6 is an advertised service, or remove the invalid AAAA record. Keep separate v4/v6 probes so fallback cannot hide a broken family.

The IPv4 probe succeeds but the IPv6 probe always times out.

Likely causeThe endpoint advertises AAAA without listener/routing support, the external prober lacks IPv6, or security policy blocks the source prefix.

Safe checks
  • Resolve AAAA and test TCP reachability from an approved external IPv6 vantage point.
  • Inspect provider/network firewall logs for the exact prober source.

ResolutionRestore end-to-end IPv6 or remove the AAAA advertisement. Do not report aggregate uptime as healthy while an advertised address family is unavailable.

TLS verification fails with a hostname mismatch.

Likely causeThe target was configured as an IP, SNI is absent/wrong, a redirect changed host, or the certificate SAN does not contain the public hostname.

Safe checks
  • Probe the DNS hostname and inspect the final URL, SNI and certificate SAN through a bounded debug request.
  • Compare the certificate presented for IPv4 and IPv6 from each external vantage point.

ResolutionUse the public hostname or an explicit reviewed `server_name` for an IP-only module, correct certificate/redirect configuration and keep `insecure_skip_verify` false.

`probe_ssl_earliest_cert_expiry` is missing.

Likely causeThe final response is plaintext, TLS failed before a certificate was observed, a redirect/module does not require TLS, or the selected prober does not expose a TLS session.

Safe checks
  • Inspect `probe_http_ssl`, status/redirect metrics and bounded debug output.
  • Confirm the target begins with HTTPS and the module has `fail_if_not_ssl: true`.

ResolutionCorrect target/module and TLS path; alert separately on `probe_success == 0` so an absent expiry series is not interpreted as a long-lived certificate.

Certificate-expiry alert fires for an old certificate after rotation.

Likely causeOne load-balancer node, CDN edge, IPv6 listener or alternate SNI still serves the old chain, or a scrape is stale.

Safe checks
  • Probe all approved vantages and both address families, grouping expiry by instance/prober.
  • Inspect current scrape timestamp and certificate serial/chain outside Prometheus using approved TLS diagnostics.

ResolutionComplete certificate rollout on every edge and verify the earliest certificate across the full path; do not silence based on a single healthy node.

HTTP status is 200 but the application is unusable.

Likely causeThe endpoint returns a generic error/login/cache page, readiness check is shallow, or body validation was omitted.

Safe checks
  • Inspect a small bounded response from the synthetic health endpoint and compare stable nonsecret content.
  • Check phase timing, redirects and backend health independently.

ResolutionUse a purpose-built unauthenticated or synthetic authenticated health endpoint and bounded header/body CEL/regex validation without downloading large pages.

The probe downloads an unexpectedly large body and consumes memory/bandwidth.

Likely causeThe target changed, compression expanded heavily, `body_size_limit` is zero, or redirects reached a download endpoint.

Safe checks
  • Inspect content length, redirect chain and configured body limit using a safe HEAD or bounded request.
  • Review the approved target URL and recent change history.

ResolutionSet a finite body limit, restrict redirects and probe a dedicated lightweight endpoint. Treat decompression and redirects as part of the resource budget.

A public caller can make the exporter probe arbitrary internal addresses.

Likely causeThe `/probe` endpoint is exposed or authenticated too broadly, and Blackbox Exporter's caller-supplied `target` behavior is being mistaken for an allowlist.

Safe checks
  • Immediately restrict access and review gateway/exporter audit logs for private, loopback, link-local and metadata destinations.
  • Verify network egress denies every destination outside the approved probe CIDRs/ports even after DNS resolution.

ResolutionTreat as an SSRF/pivot incident, isolate the exporter, investigate access, enforce Prometheus-only access plus network-layer egress allowlists and separate trust-zone probers.

A hostname passed the allowlist but resolves to a private or metadata IP.

Likely causeValidation checked only the URL string, DNS changed after review, CNAME/rebinding crossed a boundary, or redirects reached a forbidden address.

Safe checks
  • Resolve all A/AAAA/CNAME answers from the prober and inspect every redirect destination.
  • Test firewall denial for loopback, RFC1918, link-local, cloud metadata and unapproved internal prefixes.

ResolutionEnforce destination policy at the network layer after resolution and on redirects, use dedicated DNS/egress controls and split probers by trust zone.

Prometheus shows the exporter as UP while all target probes fail.

Likely causeThe `/metrics` scrape only checks exporter process health; `up` for that job does not represent `probe_success` for targets.

Safe checks
  • Inspect both exporter self-metrics and target `/probe` series.
  • Query `probe_success` and scrape age per instance/module/prober.

ResolutionCreate separate jobs, dashboards and alerts for exporter health versus target probe success, and alert when expected probe series disappear.

Prometheus probes the exporter address as the target.

Likely causeRelabeling copied or replaced `__address__` in the wrong order, or `__param_target` was never set.

Safe checks
  • Inspect discovered labels and the final scrape URL in Prometheus service discovery.
  • Compare each relabel step with the multi-target exporter pattern.

ResolutionCopy original address to `__param_target`, copy that to `instance`, then replace `__address__` with the exporter; retain a config regression test.

Target labels explode in cardinality.

Likely causeQuery strings, paths, dynamic tenant IDs or full URLs are used as `instance`, or discovery creates ephemeral combinations.

Safe checks
  • Count unique `instance`, `module`, `prober` and service labels over time.
  • Inspect sanitized target definitions for dynamic parameters.

ResolutionUse stable target IDs as labels and pass a controlled URL separately, strip query secrets, and never probe per-user/per-request targets.

A probe times out just below the Prometheus scrape timeout.

Likely causeThe module timeout leaves no scrape/serialization margin, redirects or DNS consume the budget, or the target is slow.

Safe checks
  • Compare `probe_duration_seconds` phases with module and scrape timeouts.
  • Run the same target from another approved prober to separate target from vantage congestion.

ResolutionSet module timeout below scrape timeout with explicit margin, reduce redirect/body work and alert on latency before hard failure.

ICMP works only when the exporter runs as root.

Likely causeThe kernel requires raw socket privilege and the service lacks the narrow capability or permitted ping group range.

Safe checks
  • Inspect kernel ping group range/capability and the exporter service account.
  • Confirm HTTP/TCP modules work without privilege.

ResolutionUse the supported unprivileged ICMP mechanism or only CAP_NET_RAW with a hardened service; never run the full exporter as root merely for ping.

Redirect probes unexpectedly leave the approved domain.

Likely causeThe target was compromised/misconfigured, an identity provider redirect is not allowlisted, or unrestricted `follow_redirects` crossed the egress boundary.

Safe checks
  • Inspect the bounded redirect chain and destination DNS/IP.
  • Compare with approved redirect hosts and network audit logs.

ResolutionDisable redirects or permit only the reviewed chain through network policy; create separate probes for each public dependency.

The same endpoint succeeds from one location and fails from another.

Likely causeGeo/CDN routing, DNS view, firewall allowlist, IPv6, certificate edge or provider path differs by vantage.

Safe checks
  • Compare resolved addresses, IP protocol, TLS chain, phase timing and status per prober.
  • Inspect target/CDN logs for the distinct source networks.

ResolutionTreat prober location as a first-class label, repair the affected path and avoid aggregating all vantages into one boolean before diagnosis.

Blackbox configuration reload fails.

Likely causeYAML/module syntax, regex/CEL, referenced secret file or option differs from version 0.28.0.

Safe checks
  • Run `blackbox_exporter --config.check` with the candidate and exact binary.
  • Inspect the one reload failure without printing credentials.

ResolutionKeep the running configuration, fix the candidate offline, check it, then reload/canary one prober before fleet rollout.

Authenticated probe credentials appear in configuration or debug output.

Likely causeInline basic/bearer values were used, debug output captured headers, or support artifacts included protected files.

Safe checks
  • Restrict and scan affected artifacts without echoing values.
  • Audit credential use and every config revision.

ResolutionRotate the credential, use `password_file`, `credentials_file` or header files, minimize debug, and regenerate secret-free evidence.

Prometheus marks the probe scrape UP while `probe_success` is 0.

Likely causeThe exporter successfully returned metrics describing a failed probe; this is expected multi-target behavior.

Safe checks
  • Inspect `up`, `probe_success`, phase/status/TLS metrics together.
  • Confirm the scrape is fresh and the correct module was passed.

ResolutionAlert on `probe_success` and missing/stale series, not solely on Prometheus `up`; keep exporter self-health separate.

After the procedure

Alternatives and next steps

Consider these alternatives

  • Use provider synthetic monitoring when globally distributed maintained vantage points and browser-level checks outweigh self-hosted control; retain independent TLS and target governance.
  • Use a small purpose-built health checker with a compiled target allowlist when untrusted users must request checks; do not put Blackbox Exporter directly behind such an interface.
  • Use DNS, TCP or gRPC Blackbox modules when protocol-level availability is more meaningful than HTTP; maintain the same egress and timeout boundaries.
  • Use service-mesh/internal health metrics for component diagnosis, but keep at least one truly external blackbox signal for user-visible reachability.

Operate it safely

  • Add a third provider/region vantage for the highest-tier services, preserving a bounded `prober` label and independent DNS/egress.
  • Run monthly automated negative tests for loopback, private, link-local, metadata, unapproved port, hostile redirect and DNS-change destinations.
  • Integrate certificate-expiry ownership with the certificate issuance inventory while retaining external observed expiry as an independent control.
  • Review target registry, label cardinality, probe cost and owner/runbook freshness every 90 days; remove abandoned endpoints rather than silencing them.

Reference

Frequently asked questions

Why can `up` be 1 when the site is down?

Prometheus successfully scraped the exporter, so `up=1`. The exporter successfully reported that the remote probe failed, represented by `probe_success=0`.

Can an application allowlist make arbitrary public `/probe` safe?

Not by itself. DNS, redirects and address changes can cross boundaries after string validation. Keep the endpoint private and enforce network egress after resolution.

Should protocol fallback be enabled?

It can model a user who can fall back, but separate no-fallback IPv4 and IPv6 probes are required when both families are advertised and operated.

Recovery

Rollback

Do not roll back by making `/probe` public, widening egress, enabling insecure TLS, address-family fallback or unlimited bodies/timeouts. Preserve probe/self metrics and audit evidence, restore the previous signed atomic module/registry/Prometheus bundle to one canary location, then stage across remaining vantages.

  1. Stop the faulty target/module rollout while leaving other external vantages and unaffected targets running.
  2. Preserve service-discovery labels, exporter debug for only synthetic targets, firewall/DNS audit and alert delivery evidence.
  3. Validate the previous module and Prometheus configuration with exact binaries and confirm it still satisfies current egress/TLS policy.
  4. Restore one canary prober and rerun forbidden-destination, IPv4/IPv6, SNI, body, timeout and safe notification fixtures.
  5. If credentials leaked, revoke/rotate them and replace inline values with protected files before any restart.
  6. Complete rollout only after cross-vantage results agree, missing-series alerts clear, capacity has headroom and production paging remains untouched.

Evidence

Sources and review

Verified 2026-07-24Review due 2026-10-22
Prometheus Blackbox ExporterofficialBlackbox Exporter configurationofficialBlackbox Exporter 0.28.0 releaseofficialPrometheus multi-target exporter guideofficialPrometheus configuration relabelingofficialPrometheus alerting rulesofficial