Observability & Monitoring / Networking & DNS / Security

Monitor network devices with SNMP Exporter and Grafana

Generate narrow SNMP Exporter 0.30.1 modules from licensed MIBs, poll devices with SNMPv3 authPriv, control walk/cardinality cost and build HA Prometheus/Grafana dashboards and alerts.

300 min11 stepsHigh-impact changeRevision 2

Save or explore

#snmp #snmp-exporter #grafana #prometheus #network-devices #mib #snmpv3 #authpriv

0 of 11 steps completed

Goal

Turn legacy network telemetry into trustworthy metrics without plaintext community access, hand-edited generated files, overloaded device control planes or unsafe device strings.

Supported environments

SNMP Exporter 0.30.1
Prometheus 3.13.1
Grafana 13.1.0
Network devices SNMPv3 authPriv-capable supported firmware

Prerequisites

Inventory and ownership Stable device IDs, management targets/VRFs, firmware families, owners and operational metric use cases.
Management security SNMPv3 authPriv, read-only views, exporter source ACLs and no Internet/public management exposure.
MIB rights and toolchain Authoritative licensed MIBs, digests, pinned 0.30.1 generator and recorded Net-SNMP parser.
Lab and capacity Representative device family fixtures plus packet, duration, retry, device CPU and series budgets.
Secrets and exporter hosts Protected ephemeral credential delivery, two hardened management hosts and authenticated exporter web access.
Monitoring release path Prometheus/Grafana provisioning, safe nonpaging Alertmanager receiver, HA/dedup design and signed atomic rollback bundle.

Operating boundary

OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.

Full guide

What you will build

System

A secure SNMP monitoring plane using SNMP Exporter 0.30.1, SNMPv3 authPriv with SHA-256 authentication and AES privacy, protected runtime secret expansion and management-network ACLs that permit read-only polling only from exporter identities.
A reproducible generator workflow that pins the generator, Net-SNMP toolchain and licensed MIB inputs per device family, generates rather than hand-edits `snmp.yml`, walks only approved OIDs and proves packet, duration and series budgets before onboarding.
Prometheus 3.13.1 multi-target relabeling, redundant exporter paths and Grafana 13.1 dashboards/alerts that use stable device and interface identity, sanitize untrusted device strings and surface reachability, reboot, port state, errors, discards, utilization and scrape cost.

Observable outcome

A lab and canary device answer only SNMPv3 authPriv from approved management sources; invalid user/password/privacy/context and SNMPv2c attempts fail and are audited without exposing credentials.
Generated modules have traceable MIB licenses and digests, bounded walks/retries/timeouts, no duplicate broad tables and a representative scrape that stays within device CPU, packet, duration and Prometheus series budgets.
Two exporter paths run identical generated configuration, dashboards remain responsive with controlled variables/recording rules, and synthetic interface/reboot/failure fixtures produce accurate nonpaging alerts.

Architecture

How the parts fit together

SNMP Exporter 0.30.1 runs on at least two hardened management hosts and is reachable only by Prometheus over authenticated TLS. Its `/snmp` target/auth/module parameters come from an administrator-owned device inventory; egress permits SNMP only to managed device prefixes. Devices expose dedicated read-only SNMPv3 authPriv identities restricted by source ACL. The exporter loads generated `snmp.yml` with runtime-expanded credentials from a protected ephemeral environment; generated configuration is never hand-edited. For each device family, a pinned generator and Net-SNMP version parse only licensed required MIBs from an isolated directory and produce a signed artifact. Modules walk a minimal set of system/interface/vendor health OIDs with explicit request timeout, retries and max repetitions. Prometheus uses the multi-target pattern and separates exporter self-health from device metrics. Recording rules compute counter rates/utilization and Grafana 13.1 dashboards use stable inventory labels. Device-provided strings are untrusted display metadata, never routing identities, URLs or commands. HA exporters are stateless but must share config/secret/path parity; Prometheus HA pairs use a defined local/failover exporter path so identical series do not collide unexpectedly.

Managed devicesExpose read-only SNMPv3 authPriv on management addresses with source ACL, device audit and no Internet reachability.

SNMP Exporter 0.30.1 pairTranslate approved SNMP OIDs to Prometheus metrics while holding protected authentication profiles and enforcing bounded walks.

SNMP generatorParses pinned standard/vendor MIBs and a reviewable generator.yml to produce the runtime snmp.yml; operators never hand-edit generated output.

MIB repositoryStores or locally supplies only legally usable, versioned device-family MIBs plus licenses/digests and dependency records.

Inventory and management networkOwn stable device IDs, family/module/auth mapping and restrict exporter UDP/161 egress plus Prometheus web callers.

Prometheus 3.13.1Discovers reviewed devices, relabels target/auth/module parameters, scrapes self/device metrics and evaluates recording/alert rules.

Grafana 13.1Presents bounded inventory, interface utilization/errors/state, device health and polling cost through provisioned versioned dashboards.

Safe alert receiverAccepts lab/canary device failures without production paging and validates route/runbook behavior.

Inventory owner registers a management address, stable device ID, device family, supported firmware, SNMPv3 context and module/auth profile.
A dedicated read-only SNMPv3 authPriv identity is created on the device and restricted to exporter source addresses; secrets enter the runtime secret workflow only.
Operators obtain licensed MIBs, pin their digests/toolchain and author a narrow generator.yml per family; the generator writes a reviewed snmp.yml artifact.
A lab device is walked with bounded GETBULK, timeout and retries; packet, duration, device CPU and metric-series budgets must pass before canary.
Prometheus maps inventory target/auth/module into `/snmp` parameters and scrapes a private exporter while separate `/metrics` job monitors exporter health.
Recording rules calculate rates/utilization and sanitize inventory labels; Grafana dashboards and alerts use stable IDs and display device strings only as untrusted bounded context.
Both exporter paths receive one signed artifact and coordinated secrets; failure/canary/rollback tests prove parity, alert accuracy and device safety.

Assumptions

SNMP Exporter 0.30.1, Prometheus 3.13.1, and Grafana 13.1.0 are the reviewed monitoring and visualization releases as of 2026-07-28.
All production devices support SNMPv3 authPriv with an approved modern authentication/privacy combination such as SHA-256 and AES. Legacy SNMPv1/v2c exceptions require separate documented risk and isolated networks and are not the baseline.
Management addresses and SNMP are not Internet-accessible. Device ACLs permit UDP/161 only from dedicated exporter addresses; exporter egress reaches only managed device CIDRs.
Prometheus is the only routine caller of exporter `/snmp` and `/metrics`; exporter-toolkit TLS/auth protects the HTTP endpoint across hosts.
Device inventory supplies stable IDs, management targets, family, module, auth and owner. End users cannot select an arbitrary target, auth profile, context or module.
A representative lab device exists for every firmware/device family. The team can observe control-plane CPU, packet rate and SNMP audit during generator and load testing.
Vendor MIB license and redistribution terms are reviewed. Restricted MIBs are supplied locally by authorized operators and are never copied into public repositories or artifacts contrary to license.
Generated snmp.yml is reproducible from pinned generator.yml, MIB digests, generator version and Net-SNMP parser options. Generated output is not manually edited.
SNMPv3 secrets are delivered through a protected ephemeral runtime environment, are absent from Git, argv, generated evidence, Grafana and Prometheus labels, and are rotated with device configuration.
HA Prometheus/exporter topology, replica labeling and remote-storage deduplication are defined. Two exporters do not accidentally emit indistinguishable duplicate scrapes into one local Prometheus.
Interface alias/description/name and vendor strings are untrusted mutable input. Numeric/stable inventory identity remains available when strings change or contain hostile characters.

Key concepts

MIB and OID: A Management Information Base describes named objects in the numeric SNMP OID tree. Runtime polling uses generated numeric mappings; possession of a vendor MIB may be license-restricted.
SNMPv3 authPriv: SNMPv3 security level providing message authentication/integrity and privacy encryption. Both auth and privacy credentials/protocols must match the device engine/context.
Generator: The official tool that combines generator.yml instructions and MIB definitions into runtime snmp.yml. The generated file is an artifact, not the hand-edited source.
GETBULK and max_repetitions: SNMPv2/v3 operation retrieving multiple successive objects per request. Larger repetitions reduce round trips but can overload or exceed buggy agent limits.
Lookup: A generated mapping that attaches another table column such as ifName or ifAlias to metrics indexed by ifIndex. It adds useful context and cardinality/cost.
Counter64: High-capacity SNMP counter suited to fast interfaces. Prometheus queries use rate/increase and account for resets; exporter wrapping handles float precision boundaries.
EnumAsStateSet: Generator override representing each possible state as labeled series. It is useful for alerts but multiplies series by enum values.
Multi-module scrape: One request asks exporter to run several modules, possibly concurrently. Walks are not deduplicated, so overlapping modules multiply device load.
Untrusted device string: Mutable values such as sysName, sysDescr, ifAlias and ifDescr controlled by device administrators or compromised firmware; safe only as bounded escaped display context.

Before you copy

Values used in this guide

{{snmpExporterVersion}}

Pinned SNMP Exporter and generator version.

Example: 0.30.1

{{prometheusVersion}}

Approved Prometheus 3.13.1 release.

Example: 3.13.1

{{grafanaVersion}}

Approved Grafana 13.1 patch.

Example: 13.1.0

{{snmpGeneratorConfig}}

Per-family generator source path.

Example: /srv/snmp-generator/campus-switch/generator.yml

{{snmpGeneratedConfig}}

Generated runtime configuration path.

Example: /etc/snmp_exporter/snmp.yml

{{snmpMibDir}}

Licensed isolated MIB directory for one family.

Example: /srv/snmp-generator/campus-switch/mibs

{{snmpExporterAddress}}

Private exporter address used by Prometheus.

Example: snmp-exporter-a.internal.example:9116

{{snmpDevice}}

Approved management address from inventory.

Example: 10.70.20.11

{{snmpDeviceId}}

Stable inventory identifier.

Example: core-sw-01

{{snmpModule}}

Generated family module.

Example: campus_switch

{{snmpAuth}}

Named protected auth profile.

Example: campus_v3

{{snmpWebConfig}}

Exporter-toolkit TLS/auth file.

Example: /etc/snmp_exporter/web.yml

{{snmpUsername}}secret

Runtime SNMPv3 username environment value.

Example: prometheus-ro

{{snmpAuthPassword}}secret

Runtime SNMPv3 authentication password.

Example: provided-by-secret-manager

{{snmpPrivPassword}}secret

Runtime SNMPv3 privacy password.

Example: provided-by-secret-manager

Security and production boundaries

SNMPv3 authPriv is the production baseline. SNMPv1/v2c communities travel without encryption and are not treated as secret transport; do not downgrade for convenience.
The exporter contains reusable credentials and accepts target/auth/module request parameters. Protect its HTTP endpoints, restrict callers and enforce SNMP egress only to inventory-managed prefixes.
Runtime environment expansion avoids Git/argv but environment can still leak through privileged process inspection, crash dumps and support bundles. Use dedicated identity, disable dumps and rotate on exposure.
Device strings are untrusted. Escape/limit them for display, never construct URLs, routing identities, Markdown, SQL or shell commands from them, and never promote unbounded strings to labels.
Vendor MIB licensing is a security/supply-chain and legal boundary: obtain from authoritative source, verify digest, isolate by family and do not redistribute without permission.

Stop before continuing if

Stop if production uses SNMPv1/v2c without an approved exception, authPriv credentials appear in source/output or device ACL permits untrusted source networks.
Stop if `/snmp` accepts arbitrary callers/targets/auths/modules, exporter egress reaches outside managed device CIDRs or management SNMP is Internet-visible.
Stop if generated output was hand-edited, MIB provenance/license/digests are unknown, generator toolchain is unpinned or device-family collisions are unresolved.
Stop onboarding if walk packets/duration/device CPU/series exceed budget, scrape timeout lacks margin, multi-module duplication exists or untrusted strings drive cardinality.
Stop rollout if HA exporters differ, dashboards/alerts misuse interface state/counter resets, safe fixtures can page production or secret rotation lacks rollback.

decision

Inventory device families, owners and metric budgets

read-only

Register stable device ID, management address/VRF, owner, firmware/family, SNMPv3 support, context, expected interface/table scale, selected metrics and SLO. Set maximum packets, duration, retries, device CPU impact and Prometheus series per scrape before importing MIBs.

Why this step matters

SNMP tables can turn one device into hundreds of thousands of series and control-plane requests. Inventory and budgets keep collection proportional to an operational question.

What to understand

Estimate interfaces, VLANs, routes, neighbors, clients, sensors and enum state expansion for each family.

Choose stable inventory labels and mark sysName/ifAlias/ifDescr as untrusted display values.

Record firmware and lab representative because agents vary materially within one vendor.

System changes

Creates an inventory and cost contract; devices remain unchanged.

Syntax explained

series<=5000: Example per-device budget to be replaced by measured approved capacity.

Command

Fill variables0/4 ready

Values stay on this page and are never sent or saved.

Snmp Device Id{{snmpDeviceId}}Stable inventory identifier.Snmp Device{{snmpDevice}}Approved management address from inventory.Snmp Module{{snmpModule}}Generated family module.Snmp Auth{{snmpAuth}}Named protected auth profile.

printf '%s\n' '{{snmpDeviceId}}|{{snmpDevice}}|{{snmpModule}}|{{snmpAuth}}|series<=5000|duration<10s' | sha256sum

Example output / evidence

cbbe1378ac715086b22552323b0502724535179cb896e1cf4f1f8df55d3637f1  -

Checkpoint: Verify this layer before continuing

Continue whenEach family has owner, lab fixture, stable identity, SNMPv3 support and packet/duration/CPU/series budgets.

Stop whenStop if inventory is user-controlled, owner/firmware is unknown or requested tables have no operational use.

If this step fails

A request proposes full route and MAC tables for every switch.

Likely causeCollection was chosen from available MIBs rather than an incident/use case.

Safe checks

Estimate rows and labels.
Map each table to dashboard/alert owner.

ResolutionRemove unused high-cardinality tables and begin with system/IF-MIB health.

Security notes

Management addresses and device descriptions are operator-only inventory data.

Alternatives

Use vendor streaming telemetry for high-volume dynamic tables.

Stop conditions

Stop if inventory is user-controlled, owner/firmware is unknown or requested tables have no operational use.

command

Verify exporter, generator, Prometheus and Grafana versions

read-only

Pin SNMP Exporter/generator 0.30.1, Prometheus/promtool 3.13.1 and Grafana 13.1.0 from official immutable artifacts. Record checksums, Net-SNMP parser version and release/security review so regeneration is reproducible.

Why this step matters

Generated OID mappings depend on exporter/generator and Net-SNMP parsing. Exact versions turn configuration into a reproducible supply-chain artifact.

What to understand

Verify checksums/signatures before execution and retain immutable source/container digest.

Record Net-SNMP version and mibopts with each generated artifact.

Use current reviewed security patches without silently regenerating all families.

System changes

Records tool identity; no device or service changes.

Syntax explained

--version: Prints exact executable build identity for artifact and generator parity.

Command

snmp_exporter --version && ./generator --version && prometheus --version && promtool --version

Example output / evidence

snmp_exporter, version 0.30.1
generator, version 0.30.1
prometheus, version 3.13.1
promtool, version 3.13.1
Grafana API health: version=13.1.0 database=ok

Checkpoint: Verify this layer before continuing

Continue whenExporter/generator match 0.30.1, monitoring/UI versions are approved and parser toolchain is recorded.

Stop whenStop on floating/unverifiable artifact, mismatched generator/exporter or unrecorded parser version.

If this step fails

A regenerated file differs with unchanged generator.yml.

Likely causeMIBs, generator or Net-SNMP parsing environment changed.

Safe checks

Compare tool/MIB digests.
Review mibopts and environment.

ResolutionRebuild in pinned generation environment and treat diff as reviewed migration.

Security notes

Do not run unverified vendor MIB installers or generator images.

Alternatives

Use a hermetic approved container for generator plus locally mounted licensed MIBs.

Stop conditions

Stop on floating/unverifiable artifact, mismatched generator/exporter or unrecorded parser version.

config

Create a read-only SNMPv3 authPriv boundary

danger

On each device create a dedicated read-only SNMPv3 user using SHA-256 authentication and AES privacy, optional context and an ACL limited to exporter source addresses. Disable or remove unused v1/v2c and write-capable access. Deliver credentials through the protected runtime secret workflow.

Why this step matters

authPriv protects credential integrity and response privacy, while source ACL and read-only view reduce stolen-credential and exporter compromise impact.

What to understand

Use a separate identity per environment/device class where rotation and attribution require it.

Confirm the vendor supports the chosen protocols and how keys localize to engine ID/context.

Create a second recovery identity before rotating the only working user; revoke old only after canary.

System changes

Changes device management authentication/ACL; mistakes can remove monitoring or grant management access.

Syntax explained

authPriv: Requires both message authentication/integrity and privacy encryption.
read-only view: Restricts the SNMP identity to GET/GETNEXT/GETBULK on approved OIDs, not SET.
source allowlist: Limits where valid credentials can be used from the management network.

File device-snmp-policy

Configuration

Fill variables0/1 ready

Values stay on this page and are never sent or saved.

Snmp Username{{snmpUsername}}Runtime SNMPv3 username environment value.

identity = "{{snmpUsername}}"
version = "3"
security_level = "authPriv"
auth_protocol = "SHA256"
priv_protocol = "AES"
access = "read-only"
source_allowlist = ["snmp-exporter-a", "snmp-exporter-b"]

Example output / evidence

Device: core-sw-01
SNMP versions accepted: v3
Security level: authPriv
Auth protocol: SHA-256
Privacy protocol: AES
View: read-only approved OIDs
Allowed sources: 10.50.20.21/32, 10.50.30.21/32
Invalid auth test: denied and audited
SNMPv2c test: denied

Checkpoint: Verify this layer before continuing

Continue whenApproved exporter succeeds read-only; invalid password/source, v2c and SET all fail/audit.

Stop whenStop if no recovery identity/console, device supports only deprecated crypto, write access exists or secrets enter CLI history.

If this step fails

New user authenticates but cannot decrypt responses.

Likely causePrivacy protocol/password or engine/context localization differs.

Safe checks

Inspect nonsecret protocol/context and device audit.
Test one sysUpTime GET.

ResolutionCorrect authPriv profile or coordinate rotation; do not downgrade security level.

Security notes

Use vendor's protected secret entry mechanism; never show auth/priv passwords in screenshots.

Alternatives

Isolate a legacy v2c device behind a risk-approved collector/network until replacement.

Stop conditions

Stop if no recovery identity/console, device supports only deprecated crypto, write access exists or secrets enter CLI history.

warning

Acquire and isolate licensed MIB inputs

caution

Obtain standard and vendor MIBs from authoritative sources or the device vendor, review license/redistribution terms, record version and SHA-256 digest, scan as text and keep each family in an isolated directory with only required dependencies. Do not copy restricted MIBs to a public repository.

Why this step matters

MIBs are code-like parser inputs and copyrighted material. Provenance, isolation and license review prevent namespace collisions, malicious inputs and unauthorized redistribution.

What to understand

Keep one directory per family as official generator recommends and include only dependency closure.

Record acquisition URL/date/license, firmware applicability and digest.

Generated output may be distributable differently from the MIB; legal review decides, not assumption.

System changes

Stores licensed text inputs and manifests; improper handling can create legal or parser/supply-chain risk.

Syntax explained

find ... -print0: Handles filenames safely while enumerating the exact family directory.
sort -z: Produces deterministic manifest ordering without breaking on spaces.
sha256sum: Detects input changes; source/license records establish provenance.

Command

Fill variables0/1 ready

Values stay on this page and are never sent or saved.

Snmp Mib Dir{{snmpMibDir}}Licensed isolated MIB directory for one family.

find {{snmpMibDir}} -type f -print0 | sort -z | xargs -0 sha256sum

Example output / evidence

a7d705e3a194b0eae49707df6c3e7e1a8cfd3e70af499bb03342a482c31aef64  IF-MIB
1069ab126d95b1c8eab5fd2ad296badaaff9e689a03743540622974e1fbdd45a  SNMPv2-MIB
38b6a9f3f32631227f347309a8e6ad39a76c5734dd5f44f06e40bc8651ff352f  VENDOR-HEALTH-MIB
License review: approved for internal generation; redistribution=false
Unexpected executable/binary files: 0

Checkpoint: Verify this layer before continuing

Continue whenEvery MIB is text, required, licensed/provenanced, digest-pinned and isolated by family.

Stop whenStop on unknown source/license, binary/executable content, namespace collision or redistribution uncertainty.

If this step fails

Generator resolves one symbol from the wrong vendor MIB.

Likely causeFamily directories were combined and duplicate symbols collide.

Safe checks

Inspect fully qualified symbol and import tree.
Compare directory/digests.

ResolutionSeparate families and qualify MIB::symbol; regenerate under pinned toolchain.

Security notes

Vendor portals/MIBs can include confidential metadata; restrict repository and honor license.

Alternatives

Use numeric OIDs/default official modules where legal MIB input cannot be retained.

Stop conditions

Stop on unknown source/license, binary/executable content, namespace collision or redistribution uncertainty.

config

Author generator.yml with narrow walks and auth placeholders

caution

Define SNMPv3 auth using runtime environment placeholders and a device-family module that walks only system uptime plus approved interface/high-capacity counters and health OIDs. Set explicit max repetitions, retries and per-request timeout. Add only necessary lookups and filters; avoid broad route/MAC/client trees.

Why this step matters

generator.yml is the reviewable source of truth. Narrow OIDs and explicit request behavior protect low-powered device control planes and Prometheus cardinality.

What to understand

Count lookups: each string becomes a label on dependent metrics and can be mutable/untrusted.

Use high-capacity counters for fast interfaces and stable ifIndex plus inventory for identity.

One retry and two-second per-request timeout still multiply across a walk; measure total worst-case.

System changes

Defines generated metrics and secret placeholders; broad changes increase device load and data exposure.

Syntax explained

security_level: authPriv: Selects authenticated and encrypted SNMPv3 requests.
max_repetitions: 20: Bounds objects requested per GETBULK response for this tested family.
retries: 1: Limits repeated device load and worst-case scrape time.
lookups: Adds useful interface display values while increasing label/cardinality sensitivity.

File {{snmpGeneratorConfig}}

Configuration

auths:
  campus_v3:
    version: 3
    security_level: authPriv
    username: ${SNMP_USERNAME}
    password: ${SNMP_AUTH_PASSWORD}
    auth_protocol: SHA256
    priv_protocol: AES
    priv_password: ${SNMP_PRIV_PASSWORD}
modules:
  campus_switch:
    walk:
      - SNMPv2-MIB::sysUpTime
      - IF-MIB::ifAdminStatus
      - IF-MIB::ifOperStatus
      - IF-MIB::ifInErrors
      - IF-MIB::ifOutErrors
      - IF-MIB::ifInDiscards
      - IF-MIB::ifOutDiscards
      - IF-MIB::ifHCInOctets
      - IF-MIB::ifHCOutOctets
      - IF-MIB::ifHighSpeed
    max_repetitions: 20
    retries: 1
    timeout: 2s
    lookups:
      - source_indexes: [ifIndex]
        lookup: IF-MIB::ifName
      - source_indexes: [ifIndex]
        lookup: IF-MIB::ifAlias

Example output / evidence

Auth profiles: campus_v3 (version=3, security=authPriv)
Modules: campus_switch
Walk roots: 4
Lookups: ifName, ifAlias
Max repetitions: 20
Retries: 1
Per-request timeout: 2s
Estimated series on lab device: 1842

Checkpoint: Verify this layer before continuing

Continue whenSource contains only approved OIDs, bounded walk settings and runtime secret placeholders; series estimate fits budget.

Stop whenStop if real secret appears, broad subtree lacks use, worst-case exceeds scrape timeout or untrusted lookup drives routing.

If this step fails

Estimated series exceeds budget after adding a vendor table.

Likely causeTable index creates one series per client/session and enum expansion multiplies it.

Safe checks

Count lab rows and states.
Map each metric to a panel/alert.

ResolutionRemove/filter the table or collect it separately at lower frequency after owner approval.

Security notes

Environment placeholders remain secret at runtime; generator source is still operator-sensitive.

Alternatives

Use official default if_mib module when custom MIBs are unnecessary.

Stop conditions

Stop if real secret appears, broad subtree lacks use, worst-case exceeds scrape timeout or untrusted lookup drives routing.

command

Generate snmp.yml reproducibly instead of hand-editing

caution

Run the pinned 0.30.1 generator in the isolated family directory with exact MIB paths and generator.yml, writing a candidate snmp.yml. Record command, tool/MIB/source/output digests and diff generated OIDs/metrics against the prior revision. Never patch generated YAML by hand.

Why this step matters

Generation resolves OID types, indexes, enums and lookups consistently. Artifact diff exposes accidental broad walks or parser changes before runtime.

What to understand

Use separate output per family, validate, then merge only unique auth/module names as official workflow describes.

Inspect generated numeric walk/get lists, metric types/help and lookup labels, not only file checksum.

Commit generator.yml and permissible manifests/output; restricted MIBs follow license policy.

System changes

Writes/replaces candidate generated runtime configuration; a bad artifact can change every scrape.

Syntax explained

-m: Selects only the isolated licensed family MIB directory.
-g: Uses the reviewable generator source.
-o: Writes generated runtime artifact; it is not an editable source file.
--dry-run: Makes SNMP Exporter 0.30.1 parse the generated artifact and exit without opening a listener or polling a device; only explicitly synthetic validation secrets are used.

Command

Fill variables0/3 ready

Values stay on this page and are never sent or saved.

Snmp Mib Dir{{snmpMibDir}}Licensed isolated MIB directory for one family.Snmp Generator Config{{snmpGeneratorConfig}}Per-family generator source path.Snmp Generated Config{{snmpGeneratedConfig}}Generated runtime configuration path.

./generator generate -m {{snmpMibDir}} -g {{snmpGeneratorConfig}} -o {{snmpGeneratedConfig}} && SNMP_USERNAME=validation-user SNMP_AUTH_PASSWORD=synthetic-auth-only SNMP_PRIV_PASSWORD=synthetic-priv-only snmp_exporter --dry-run --config.file={{snmpGeneratedConfig}} --config.expand-environment-variables && sha256sum {{snmpGeneratorConfig}} {{snmpGeneratedConfig}}

Example output / evidence

level=info msg="Loading MIBs" path=/srv/snmp-generator/campus-switch/mibs
level=info msg="Generating config" module=campus_switch
level=info msg="Generated metrics" count=28
level=info msg="Written config" path=/etc/snmp_exporter/snmp.yml
level=info msg="Configuration parsed successfully"
4b0e0f5c6da415afc43bcf6fab6d554c5aed3c1e26d22fe3f231b538471bed93  generator.yml
c7ed0116b1789e798b38af658f3baffea220d61ef5ec8c414a8d2ccba5e3d4e9  snmp.yml
Manual generated-file edits: 0

Checkpoint: Verify this layer before continuing

Continue whenReproduction yields identical digest; reviewed diff contains only intended OIDs/types/lookups and no secrets.

Stop whenStop on nondeterministic output, duplicate names, unexpected MIB/OID/type, real secret or unlicensed artifact.

If this step fails

Two runs produce different snmp.yml.

Likely causeToolchain/MIB order/version or environment changed.

Safe checks

Compare tool/MIB/source digests.
Run in clean pinned generation environment.

ResolutionPin missing input and regenerate until deterministic; review legitimate version migrations.

Security notes

Generated auth section must retain placeholders, never resolved credential values.

Alternatives

Use the official shipped snmp.yml unmodified for supported modules.

Stop conditions

Stop on nondeterministic output, duplicate names, unexpected MIB/OID/type, real secret or unlicensed artifact.

verification

Canary the generated module against a lab device

read-only

Start a canary exporter on loopback with protected synthetic/runtime credentials and generated config. Scrape one representative lab device and verify metric names/types/labels, authPriv, packets/retries, duration, device CPU and series count. Exercise invalid auth and SNMPv2c denial.

Why this step matters

A syntactically generated configuration can still overload or misinterpret a real agent. Lab/canary evidence validates both sides before fleet polling.

What to understand

Use a representative interface/table scale and firmware; a tiny emulator is insufficient for capacity.

Inspect raw metrics for expected types and stable indexes, including hostile/malformed string fixtures if lab supports them.

Compare packets, retries, duration and device CPU against budgets at intended scrape interval.

System changes

Runs bounded read-only SNMP requests against the approved lab device.

Syntax explained

target: Approved lab management address; endpoint is private and cannot accept arbitrary callers.
auth: Named authPriv profile; secrets remain inside exporter process.
module: Generated bounded OID/lookup plan.

Command

Fill variables0/3 ready

Values stay on this page and are never sent or saved.

Snmp Device{{snmpDevice}}Approved management address from inventory.Snmp Auth{{snmpAuth}}Named protected auth profile.Snmp Module{{snmpModule}}Generated family module.

curl --fail --silent 'http://127.0.0.1:19116/snmp?target={{snmpDevice}}&auth={{snmpAuth}}&module={{snmpModule}}' | grep -E '^(sysUpTime|ifHCInOctets|ifOperStatus|snmp_scrape_)' | head -20

Example output / evidence

sysUpTime 3.812734e+08
ifHCInOctets{ifAlias="core-uplink",ifIndex="49",ifName="Ethernet49"} 8.532190441e+12
ifOperStatus{ifAlias="core-uplink",ifIndex="49",ifName="Ethernet49"} 1
snmp_scrape_duration_seconds{module="campus_switch"} 0.842
snmp_scrape_packets_sent{module="campus_switch"} 18
snmp_scrape_packets_retried{module="campus_switch"} 0
Series: 1842
Device control-plane CPU delta: +0.4%
Invalid auth: denied
SNMPv2c: denied

Checkpoint: Verify this layer before continuing

Continue whenMetrics/types/labels are correct, auth negative tests fail and all packet/duration/CPU/series budgets pass.

Stop whenStop if target is production without approval, write access works, secret appears, device CPU/retries/duration/series exceeds budget.

If this step fails

Scrape succeeds but ifHC counters are absent.

Likely causeDevice/MIB lacks high-capacity counters, generator filtered them or table access view excludes OIDs.

Safe checks

Run bounded GET for exact OIDs.
Inspect generated metric list and SNMP view.

ResolutionCorrect read-only view/module for supported counters or document safe 32-bit limitations; do not fabricate metrics.

Security notes

Device strings in output are untrusted and must not be pasted into commands/URLs.

Alternatives

Use a vendor-supported simulator only for parser tests, plus one real canary for load/behavior.

Stop conditions

Stop if target is production without approval, write access works, secret appears, device CPU/retries/duration/series exceeds budget.

config

Harden and duplicate the exporter service

caution

Run two stateless exporters across management failure domains as dedicated non-root services. Load identical generated config digest and protected ephemeral SNMPv3 environment; bind private authenticated TLS web endpoints and restrict UDP/161 egress to inventory CIDRs. Keep support bundles and core dumps from exposing environment.

Why this step matters

Exporter compromise exposes reusable device credentials and management reach. Hardening and two equivalent paths reduce blast radius and maintenance outage.

What to understand

EnvironmentFile lives on protected ephemeral storage, is generated by the secret workflow and removed/rotated without entering repository.

Disable core dumps and environment collection in support bundles; root can still inspect processes, so host admin is trusted.

Both exporters need config, secret and network parity, but one Prometheus scrape path must avoid duplicate local series.

System changes

Starts authenticated services containing device credentials and management egress; mistakes can leak secrets or duplicate monitoring.

Syntax explained

--config.expand-environment-variables: Resolves only supported auth fields at runtime from protected environment.
--web.config.file: Applies TLS/authentication to `/metrics`, `/snmp` and web UI.
CapabilityBoundingSet=: Drops capabilities because SNMP UDP polling needs no privilege.

File /etc/systemd/system/snmp_exporter.service

Configuration

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

Snmp Generated Config{{snmpGeneratedConfig}}Generated runtime configuration path.Snmp Web Config{{snmpWebConfig}}Exporter-toolkit TLS/auth file.

[Service]
User=snmp-exporter
Group=snmp-exporter
EnvironmentFile=/run/snmp-exporter/credentials.env
ExecStart=/usr/local/bin/snmp_exporter --config.file={{snmpGeneratedConfig}} --config.expand-environment-variables --web.config.file={{snmpWebConfig}} --web.listen-address=10.50.20.21:9116
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
CapabilityBoundingSet=
RestrictAddressFamilies=AF_INET AF_INET6

Command

Fill variables0/1 ready

Values stay on this page and are never sent or saved.

Snmp Generated Config{{snmpGeneratedConfig}}Generated runtime configuration path.

sha256sum {{snmpGeneratedConfig}} && systemd-analyze security snmp_exporter.service --no-pager

Example output / evidence

c7ed0116b1789e798b38af658f3baffea220d61ef5ec8c414a8d2ccba5e3d4e9  /etc/snmp_exporter/snmp.yml
exporter-a config digest: MATCH
exporter-b config digest: MATCH
SNMP credential revision: v7 / v7
Public HTTP reachability: DENIED
Noninventory UDP/161 reachability: DENIED
Overall exposure level: 2.3 OK

Checkpoint: Verify this layer before continuing

Continue whenBoth private services match artifact/secret revision, run non-root and reach only inventory device prefixes.

Stop whenStop if environment leaks, config digests differ, public endpoint exists, noninventory target works or services run privileged.

If this step fails

Exporter B authenticates differently from A.

Likely causeSecret revision or environment expansion differs despite same generated file.

Safe checks

Compare nonsecret auth profile and secret revision.
Canary one lab device from both.

ResolutionSynchronize secret deployment, restart canary and verify exact metric parity before HA use.

Security notes

Process environment remains secret; limit host admin and disable dumps/support capture.

Alternatives

Use separate per-exporter SNMPv3 users for attribution if device scale supports coordinated rotation.

Stop conditions

Stop if environment leaks, config digests differ, public endpoint exists, noninventory target works or services run privileged.

config

Configure inventory-driven Prometheus relabeling

caution

Create separate exporter-self and device jobs. Inventory supplies only approved management target, stable device ID, module and auth. Relabel originals into `__param_target`, `__param_module` and `__param_auth`, use device ID as `instance`, and replace scrape address with the exporter. Set total timeout above measured module worst case with margin.

Why this step matters

The multi-target pattern keeps device identity while Prometheus actually scrapes the exporter. Inventory-only parameters prevent arbitrary target/auth/module use.

What to understand

Keep auth name nonsecret but restricted; it reveals device class/identity mapping.

Separate self job means exporter process health is not confused with device SNMP success.

HA Prometheus pairs should use defined local exporter endpoints and external replica deduplication, not duplicate exporters in one job.

System changes

Adds periodic SNMP load and new metrics; wrong mapping can query unintended devices or duplicate series.

Syntax explained

__param_target: Becomes the management target passed to `/snmp`.
__param_auth: Selects a named protected exporter auth profile without exposing credentials.
instance=device_id: Uses stable inventory identity instead of mutable management IP or device string.

File /etc/prometheus/prometheus.yml

Configuration

Fill variables0/5 ready

Values stay on this page and are never sent or saved.

Snmp Exporter Address{{snmpExporterAddress}}Private exporter address used by Prometheus.Snmp Device{{snmpDevice}}Approved management address from inventory.Snmp Device Id{{snmpDeviceId}}Stable inventory identifier.Snmp Module{{snmpModule}}Generated family module.Snmp Auth{{snmpAuth}}Named protected auth profile.

scrape_configs:
  - job_name: snmp-exporter
    scheme: https
    static_configs:
      - targets: [{{snmpExporterAddress}}]
  - job_name: snmp-devices
    metrics_path: /snmp
    scrape_interval: 60s
    scrape_timeout: 12s
    static_configs:
      - targets: [{{snmpDevice}}]
        labels:
          device_id: {{snmpDeviceId}}
          module: {{snmpModule}}
          auth: {{snmpAuth}}
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [module]
        target_label: __param_module
      - source_labels: [auth]
        target_label: __param_auth
      - source_labels: [device_id]
        target_label: instance
      - target_label: __address__
        replacement: {{snmpExporterAddress}}

Command

promtool check config /etc/prometheus/prometheus.yml

Example output / evidence

Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax
Final device scrape:
  target=10.70.20.11
  auth=campus_v3
  module=campus_switch
  instance=core-sw-01
  exporter=snmp-exporter-a.internal.example:9116
  scrape_timeout=12s

Checkpoint: Verify this layer before continuing

Continue whenpromtool validates and final discovery maps exactly one inventory device to approved module/auth/exporter with timeout margin.

Stop whenStop if arbitrary discovery labels control params, URL/credentials become labels, timeout lacks margin or HA duplicates are undefined.

If this step fails

Exporter scrapes itself as an SNMP target.

Likely causeRelabel replacement occurred before original `__address__` was copied.

Safe checks

Inspect final service-discovery labels.
Compare canonical relabel order.

ResolutionRestore copy target/module/auth, stable instance, then exporter replacement and add config fixture.

Security notes

Prometheus UI is operator-only because it exposes management targets/auth profile names.

Alternatives

Use signed file service discovery generated from CMDB instead of static configs.

Stop conditions

Stop if arbitrary discovery labels control params, URL/credentials become labels, timeout lacks margin or HA duplicates are undefined.

config

Build recording rules, alerts and safe string handling

caution

Create recording rules for interface bit rates/utilization, errors and discards using Counter64 and speed. Alert on missing device scrape, device reboot, administratively-up/operationally-down interfaces, sustained errors/discards and scrape duration near timeout. Use stable device_id/ifIndex for identity; keep sanitized ifName/ifAlias as bounded display only.

Why this step matters

Raw counters/states need protocol-aware interpretation. Admin state avoids paging for intentionally disabled ports; counter reset and scrape-cost tests prevent false alarms.

What to understand

Use rate on Counter64 and clamp/guard speed edge cases. Correlate reboots through sysUpTime resets.

Alert identity uses device ID and ifIndex; mutable strings are escaped annotation context only and length-limited.

Missing device metrics may mean exporter, network, auth or device; runbook starts with self job and scrape error.

System changes

Adds recording/evaluation load and notifications; bad rules can page every interface or accept hostile text.

Syntax explained

8 * rate(ifHCInOctets[5m]): Converts octet counter rate to bits per second while handling resets.
ifAdminStatus == 1: Limits link-down alert to ports intended to be administratively up.
on (instance, ifIndex): Joins state using stable device/interface identity.

File /etc/prometheus/rules/snmp.yml

Configuration

groups:
  - name: snmp-network
    rules:
      - record: device_interface_receive_bits_per_second
        expr: 8 * rate(ifHCInOctets[5m])
      - alert: SnmpDeviceUnavailable
        expr: up{job="snmp-devices"} == 0
        for: 5m
        labels:
          severity: critical
      - alert: InterfaceOperationallyDown
        expr: ifAdminStatus == 1 and on (instance, ifIndex) ifOperStatus != 1
        for: 5m
        labels:
          severity: warning
      - alert: SnmpScrapeNearTimeout
        expr: snmp_scrape_duration_seconds > 9.6
        for: 10m
        labels:
          severity: warning

Example output / evidence

Rule tests:
  normal counter rate: PASS
  counter reset after reboot: PASS
  admin-down port: NO ALERT
  admin-up/oper-down 5m: ALERT
  interface errors sustained: ALERT
  missing device scrape: ALERT
  scrape duration > 80% timeout: WARNING
  hostile ifAlias fixture: escaped/display-only
Safe receiver notifications: 4
Production pages: 0

Checkpoint: Verify this layer before continuing

Continue whenPositive/negative/reset/string fixtures pass and safe route receives exact actionable alerts.

Stop whenStop if admin-down ports page, reset spikes, device strings route/build URLs, series cardinality or rule cost exceeds budget.

If this step fails

Every disabled access port alerts.

Likely causeRule checks operational state without admin state or enum mapping is wrong.

Safe checks

Query both status metrics.
Inspect generator type/enum fixture.

ResolutionRequire admin-up and stable join; retain disabled-port no-fire regression test.

Security notes

Untrusted interface descriptions are never placed in links, routing labels or shell commands.

Alternatives

Alert at device/uplink tier only and expose access-port issues in dashboard/ticket queue.

Stop conditions

Stop if admin-down ports page, reset spikes, device strings route/build URLs, series cardinality or rule cost exceeds budget.

verification

Provision Grafana dashboards and rehearse HA rollback

danger

Provision versioned Grafana 13.1 dashboards with device/site/family variables sourced from stable inventory labels, recording-rule panels for fleet/device/interface health and polling-cost panels. Test escaped hostile strings, bounded query ranges and viewer permissions. Fail one exporter path, verify the HA path and restore the previous generated/config/dashboard bundle.

Why this step matters

Dashboards are operational interfaces and can create expensive queries or unsafe links. HA/rollback testing proves observability survives maintenance without double-scraping or secret/config drift.

What to understand

Provision dashboards as code and bind Prometheus data source by stable UID; avoid manually imported opaque community dashboards without review.

Default to bounded time/device scope, recording rules and stable variables. Viewer roles cannot edit data sources or access exporter secrets.

Fail only the canary/approved exporter path, compare metrics/config/secret revisions and record expected short scrape gap or dedup behavior.

System changes

Deploys dashboards and performs controlled exporter failover/rollback; wrong HA routing can duplicate or lose metrics.

Syntax explained

sha256sum: Binds generated config, rule and dashboard revisions into one reviewed release manifest.
provisioning: Makes dashboard/datasource state reproducible instead of manual UI drift.

Command

Fill variables0/1 ready

Values stay on this page and are never sent or saved.

Snmp Generated Config{{snmpGeneratedConfig}}Generated runtime configuration path.

sha256sum {{snmpGeneratedConfig}} /etc/prometheus/rules/snmp.yml /etc/grafana/provisioning/dashboards/snmp-fleet.json

Example output / evidence

c7ed0116b1789e798b38af658f3baffea220d61ef5ec8c414a8d2ccba5e3d4e9  snmp.yml
5a777e8536e5083eae67b1a62f1c7d8528c80cf121f573ec5191a903558e4e4a  snmp.yml.rules
bb5888f7a5f6e56b41076c636e6a55df59cf155118f748c7bab74cb0856e7550  snmp-fleet.json
Dashboard p95 load (24h): 1.2s
Maximum panel series: 420
Hostile string fixture: escaped
Exporter A stopped; Exporter B canary: PASS
Duplicate local series: 0
Rollback bundle: PASS

Checkpoint: Verify this layer before continuing

Continue whenDashboard is fast/bounded/escaped/read-only; HA failover preserves expected series and atomic rollback passes.

Stop whenStop if failure test targets sole production path, strings create active links, viewer can edit secrets, queries overload Prometheus or duplicates appear.

If this step fails

After failover every interface has two identical series.

Likely causeBoth exporters are scraped by one Prometheus without distinct/deduplicated replica strategy.

Safe checks

Inspect series labels and scrape targets.
Review Prometheus HA/exporter mapping.

ResolutionUse one exporter path per Prometheus replica with proper external labels/dedup, or shard devices deterministically.

Security notes

Grafana links and alert annotations must escape device strings and use inventory IDs for navigation.

Alternatives

Start with official snmp-mixin and review/modify it as versioned source for your label contract.

Stop conditions

Stop if failure test targets sole production path, strings create active links, viewer can edit secrets, queries overload Prometheus or duplicates appear.

Finish line

Verification checklist

Generated artifact./generator generate -m {{snmpMibDir}} -g {{snmpGeneratorConfig}} -o {{snmpGeneratedConfig}}Pinned inputs reproduce the reviewed digest with expected OIDs/types/lookups and no resolved secrets.

SNMPv3 and scrape budget

curl --fail --silent 'https://{{snmpExporterAddress}}/snmp?target={{snmpDevice}}&auth={{snmpAuth}}&module={{snmpModule}}'

authPriv read-only scrape succeeds within packet/duration/CPU/series limits; invalid auth, v2c and SET fail.

Prometheus discoverypromtool check config /etc/prometheus/prometheus.ymlInventory target/auth/module map to one exporter and stable instance with timeout margin.

Rules and stringspromtool test rules /etc/prometheus/rules/tests/snmp.test.ymlCounter reset, admin/oper state, missing device, cost and hostile string fixtures pass without production page.

HA and dashboardsha256sum {{snmpGeneratedConfig}} /etc/grafana/provisioning/dashboards/snmp-fleet.jsonExporter paths match config/secret revision, failover creates no unintended duplicates and provisioned dashboards meet query/access budgets.

Recovery guidance

Common problems and safe checks

The scrape times out with no SNMP response.

Likely causeUDP/161 is blocked, the management address/VRF is wrong, SNMP service is disabled, source ACL excludes exporters or the device is overloaded.

Safe checks

Confirm route and firewall counters from the management network without broad port scanning.
Use a single bounded SNMPv3 GET for sysUpTime with the approved identity and compare device audit logs.

ResolutionRepair the management route, source ACL or device service and keep the exporter isolated to approved device CIDRs. Do not fall back to an Internet-accessible or plaintext community.

SNMPv3 returns `unknownUserName`, authentication failure or decryption error.

Likely causeUsername, engine localization, auth/priv password, protocol, context or device engine ID differs from the approved profile.

Safe checks

Compare nonsecret username/protocol/context metadata with device configuration and inspect device authentication audit.
Test one synthetic/read-only GET using protected secret injection without printing credentials.

ResolutionCorrect the exact SNMPv3 authPriv profile or rotate it on device/exporter together. Never downgrade to noAuthNoPriv or SNMPv2c to make the scrape pass.

The exporter starts but the requested auth or module is not found.

Likely causePrometheus passes a name absent from the loaded generated files, glob ordering differs, duplicate names invalidated load or rollout is inconsistent.

Safe checks

Inspect loaded configuration file paths and protected digest on each exporter.
Compare service-discovery `__param_auth`/`__param_module` with generated names.

ResolutionDeploy the same generated artifact and registry mapping to every exporter, remove duplicate definitions and canary before reload.

The generated configuration suddenly walks thousands of extra OIDs.

Likely causeA broad subtree/vendor MIB was added, generator dependency changed, filter was removed or a MIB symbol resolved differently.

Safe checks

Diff generator.yml, MIB inventory/digests and generated walk/get lists.
Measure packets, scrape duration and metric series against the previous fixture.

ResolutionRestore the narrow generator input, pin MIBs/toolchain and include an artifact/series budget test before deployment.

Generator fails with duplicate or unresolved MIB symbols.

Likely causeVendor and shared MIB directories collide, dependencies are missing, two revisions define the same symbol or Net-SNMP parsing options changed.

Safe checks

Generate one device family in an isolated directory with only licensed required MIBs.
Inspect parser diagnostics and MIB import dependencies.

ResolutionSeparate device-family MIB namespaces, pin revisions/toolchain and qualify symbols with MIB names; do not enable conflict-ignoring parser options blindly.

The repository cannot legally redistribute a vendor MIB.

Likely causeA downloaded MIB carries restrictive copyright/license terms or the vendor portal grants use without redistribution.

Safe checks

Review the vendor-provided license and acquisition record.
Inspect repository history/artifacts for unauthorized copies.

ResolutionRemove/quarantine the MIB from distributed artifacts, retain only generated output when permitted, document acquisition and require operators to supply the licensed file locally.

An interface counter rate becomes negative or spikes after reboot.

Likely causeThe device rebooted, ifIndex changed, a 32-bit counter wrapped, scrape gap occurred or query did not use `rate()` on the correct Counter64 metric.

Safe checks

Compare sysUpTime, ifIndex/ifName mapping, scrape timestamps and raw high-capacity counters.
Inspect whether the metric type generated as counter and whether Prometheus detected reset.

ResolutionUse ifHC counters and `rate`, correlate reset with sysUpTime and preserve stable inventory identity; add reboot/index-change fixtures.

Dashboard shows duplicate or changing interfaces.

Likely causeifAlias/ifDescr changed, ifIndex was reused after reboot, two modules export overlapping metrics or HA scrape replicas are not deduplicated.

Safe checks

Compare series labels, module list, Prometheus replica labels and device interface inventory.
Check whether multiple modules walk the same table without deduplication.

ResolutionUse one intentional interface module, stable device/ifIndex identity plus sanitized display fields, and configure HA replica deduplication in the storage/query layer.

Prometheus cardinality grows rapidly after onboarding a device family.

Likely causeA large table, MAC/route/client session index, EnumAsStateSet or user-editable string was exported as labels/series.

Safe checks

Count series by device/module/metric and inspect top label-value counts.
Compare generated OIDs and lookups with the approved metric budget.

ResolutionRemove unnecessary tables/lookups, use static/dynamic filters carefully, drop high-risk metrics and rerun cost fixtures before onboarding.

A malicious or malformed device string breaks links or pollutes notifications.

Likely causesysName, ifAlias, ifDescr or vendor string was treated as trusted and interpolated into URLs, Markdown, routing labels or shell commands.

Safe checks

Inspect raw/sanitized synthetic strings containing quotes, markup, control characters and long values.
Review Grafana links, alert annotations and relabeling for interpolation.

ResolutionTreat device strings as display-only untrusted data, normalize length/characters, never use them in command/URL/routing construction and retain numeric stable identity.

One slow device makes the Prometheus scrape exceed its timeout.

Likely causeModule walks too much, per-request timeout/retries multiply, max_repetitions is poor for the agent or multiple modules run serially.

Safe checks

Inspect `snmp_scrape_duration_seconds`, packets sent/retried and per-module walk inventory.
Test one approved OID/module with lower repetitions on a maintenance fixture.

ResolutionNarrow the module, reduce retries/timeout, tune max_repetitions for that family and keep total worst-case below scrape timeout with margin.

Increasing `--snmp.module-concurrency` overloads devices.

Likely causeModules perform duplicate walks and concurrent GETBULK requests exceed control-plane capacity; exporter concurrency was changed without a device budget.

Safe checks

Compare packets, CPU and duration on a lab/approved device before and after.
Inspect overlapping module walk roots because multi-module handling does not deduplicate them.

ResolutionReturn to safe concurrency, merge/narrow modules or schedule separately and obtain vendor/owner capacity approval.

SNMP exporter self job is UP but device metrics are absent.

Likely causeProcess health is separate from target scrape, relabeling is wrong, `/snmp` returned an error or the module/auth target was rejected.

Safe checks

Inspect exporter `up` separately from the SNMP job's `up` and scrape error/body.
Review final discovered parameters and exporter logs for the single device.

ResolutionMaintain separate self and device jobs/alerts, repair parameter mapping and alert on missing expected device series.

Prometheus sends an arbitrary target or auth profile to `/snmp`.

Likely causeExporter endpoint accepts untrusted callers or discovery labels are not constrained to the inventory registry.

Safe checks

Restrict endpoint and inspect audit/network logs for noninventory IPs, auths and modules.
Verify egress allows UDP/161 only to managed device CIDRs.

ResolutionTreat the exporter as a network proxy, allow only Prometheus, source targets/auth/modules from reviewed inventory and enforce management-network egress.

SNMPv3 secrets appear in process environment diagnostics.

Likely causeEnvironment expansion is used but service managers, crash dumps, support bundles or operator commands expose the environment.

Safe checks

Restrict diagnostics and scan captured artifacts without printing the values.
Audit which principals can read process environment or service configuration.

ResolutionRotate credentials, deliver a protected runtime environment through the secret manager, disable core dumps and exclude environment from support artifacts.

A device returns non-increasing OIDs and the walk stops.

Likely causeThe SNMP agent is buggy, table order is inconsistent or a vendor MIB implementation violates ordering.

Safe checks

Capture only the bounded OID sequence in a lab/maintenance window and compare vendor documentation.
Test a narrow GET list that avoids the broken subtree.

ResolutionPrefer explicit GET/filter or vendor fix. Enable `allow_nonincreasing_oids` only for the isolated reviewed module because it can conceal loops/bad agents.

A device responds from a different source address and the exporter ignores it.

Likely causeMultihomed device/VRF sends replies from another interface while a connected UDP socket expects the target address.

Safe checks

Use packet metadata in an approved maintenance capture to compare request destination and reply source.
Review device routing and management-source configuration.

ResolutionFix the device management source. Use `use_unconnected_udp_socket` only for that documented family and retain strict network ACLs.

Grafana dashboard is slow or times out.

Likely causePanels query raw high-cardinality interface metrics over long ranges, variables use unbounded strings or recording rules are missing.

Safe checks

Inspect query duration/series count and variable cardinality.
Compare a bounded device/time dashboard with the same panels.

ResolutionUse recording rules for rates/utilization, scoped variables, bounded defaults and controlled labels; do not hide scrape cost by extending every timeout.

Interface-down alert fires for administratively disabled ports.

Likely causeThe rule checks operational state without requiring admin state up, or generated enum mapping differs.

Safe checks

Evaluate ifAdminStatus and ifOperStatus state sets/raw values for synthetic and real disabled interfaces.
Inspect generator override and rule fixture.

ResolutionAlert only when administratively up but operationally down for the sustained period, and test both disabled and failed ports.

Two HA exporters return different metric sets.

Likely causeGenerated config/MIB/auth revisions differ, targets resolve differently, one exporter cannot reach a VRF or environment secrets map to different identities.

Safe checks

Compare config/MIB digests, loaded versions, target route and a synthetic device scrape.
Inspect module/auth names and secret versions without revealing secret values.

ResolutionDeploy one signed generated artifact and coordinated secret revision, repair path parity and canary both before failover.

After the procedure

Alternatives and next steps

Consider these alternatives

Use the official default if_mib module unchanged when it meets the supported device and cost contract; still use SNMPv3 and protected auth.
Use vendor streaming telemetry/OpenConfig when high-frequency structured metrics and event-driven updates are supported and operationally owned; SNMP may remain for inventory/legacy coverage.
Use Grafana Alloy's embedded SNMP exporter component when standardizing collectors, after verifying the embedded configuration version and secret/HA behavior.
Use a managed network-monitoring platform when MIB lifecycle, device compatibility and HA operations exceed team capacity, while retaining least-privilege device identities.

Operate it safely

Add device-family conformance fixtures and regenerate on every MIB/generator upgrade, comparing OIDs, metrics, labels, packets, duration and series budget.
Rotate SNMPv3 credentials by adding a new read-only user, updating one canary exporter path, verifying, then revoking the old user with device audit.
Build recording rules for interface utilization/errors/discards and fleet health, keeping raw strings out of routing and query URLs.
Review MIB licensing, device firmware support, inventory ownership, exporter parity and Grafana dashboard cost every 90 days.

Reference

Frequently asked questions

Why not edit snmp.yml directly?

It is generated runtime output with many OID/type/lookup relationships. Hand edits are not reproducible and disappear on regeneration; change generator.yml or MIB inputs and regenerate.

Is an SNMPv2 community string secure if it is hard to guess?

No. SNMPv1/v2c does not encrypt it or the response. Use SNMPv3 authPriv and network/source ACLs.

Can two modules be scraped together safely?

Only after proving their walks do not overlap or overload the device. Multi-module execution does not deduplicate walks and concurrency can multiply load.

Recovery

Rollback

Do not roll back by enabling SNMPv2c, weakening authPriv, exposing `/snmp`, widening device ACLs, hand-editing generated output or extending timeouts until overload is hidden. Preserve scrape/device/audit evidence and restore the last signed atomic generator/config/rule/dashboard bundle through one canary exporter.

Stop onboarding or the changed family module while leaving unaffected device families and the other exporter path running.
Preserve generated/source/MIB/tool digests, packets/retries/duration, device CPU, series counts, Prometheus discovery and safe notification evidence.
Validate license/provenance and regenerate the previous artifact from its pinned inputs rather than copying a hand-modified runtime file.
Restore one canary exporter with the coordinated secret revision and verify authPriv positive plus invalid-auth/v2c/write negative tests.
Restore Prometheus rules/dashboard provisioning and run reset, admin-down, hostile-string, query-cost and safe-route fixtures.
Complete rollout only after both exporter paths have parity, management egress/caller boundaries hold and no duplicate/missing series remain.

Evidence

Sources and review

Verified 2026-07-24Review due 2026-10-22

Prometheus SNMP Exporterofficial SNMP Exporter generatorofficial SNMP Exporter 0.30.1 releaseofficial SNMP Exporter mixinofficial Prometheus multi-target exporter guideofficial SNMPv3 User-based Security Modelofficial AES Privacy Protocol for SNMPofficial Grafana dashboard documentationofficial Grafana 13.1 release highlightsofficial