Monitor a host with Node Exporter and Grafana
Install Node Exporter as a locked-down service, restrict metrics, add Prometheus, and verify Grafana.
Collect and visualize host metrics without publishing the exporter to the internet.
- Node Exporter 1.8+
- Prometheus 2.x, 3.x
- Grafana 11+, 12+
- Monitoring server Prometheus and Grafana are reachable on a protected network.
- Firewall plan Allow TCP 9100 only from Prometheus.
- Verified release Select a Node Exporter release and verify its checksum.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A root-owned Node Exporter binary running as a non-login account under a sandboxed systemd unit and listening only on the selected management address.
- A source-restricted firewall path and validated Prometheus scrape configuration with stable labels.
- A Grafana data source, host dashboard, and actionable alert set whose complete exporter-to-notification path is tested.
- The binary checksum matches the official release manifest and the running executable path, owner, version, and flags are recorded.
- Only the Prometheus address can reach TCP 9100; an untrusted probe fails and no public wildcard listener exists.
- Prometheus reports the intended target as up=1 with fresh node metrics and expected labels.
- Grafana panels display current samples and a controlled test alert reaches the assigned notification route with a linked runbook.
Architecture
How the parts fit together
Node Exporter reads host kernel and filesystem interfaces and exposes Prometheus text metrics over HTTP. A management-network firewall allows only Prometheus to scrape the exporter. Prometheus stores time series and evaluates alert rules; Grafana queries Prometheus for dashboards while Alertmanager or Grafana's configured routing delivers actionable notifications.
- Node Exporter collects host state and serves /metrics on the management address.
- The firewall permits TCP 9100 only from the Prometheus host.
- Prometheus periodically fetches metrics, attaches stable labels, and stores samples.
- Grafana queries Prometheus for panels; alert rules evaluate recent samples.
- A firing alert follows the configured notification route to an owner and runbook.
Assumptions
- Prometheus and Grafana already run on a protected management network and have a documented authentication, backup, and access model.
- The host has one stable management address that is not directly internet-routable.
- The official Node Exporter release archive and checksum manifest were obtained over a trusted channel and the selected architecture matches the host.
- The host firewall already has a complete persistent policy; this guide inserts one reviewed rule rather than replacing the ruleset.
- Metrics may reveal host names, mounts, devices, kernel state, and workload patterns and are classified as operationally sensitive.
Key concepts
- Exporter
- A process that exposes measurements in Prometheus format; it does not store historical data.
- Scrape
- A Prometheus HTTP collection cycle that assigns a timestamp and labels to returned samples.
- Target label
- Stable context such as job, instance, or environment used to select and aggregate series.
- Cardinality
- The number of unique label combinations; uncontrolled labels can exhaust Prometheus resources.
- Recording rule
- A precomputed PromQL expression that improves repeated dashboard and alert queries.
- Actionable alert
- A condition with user impact, owner, urgency, runbook, and a tested route rather than a raw noisy metric threshold.
Before you copy
Values used in this guide
{{managementAddress}}Host address where Node Exporter listens.
Example: 10.0.10.15{{prometheusAddress}}Only source address allowed to reach TCP 9100.
Example: 10.0.10.20{{environmentLabel}}Stable low-cardinality environment label.
Example: productionSecurity and production boundaries
- Do not expose Node Exporter directly to the internet. Its metrics provide a useful infrastructure inventory even when the endpoint has no remote-control capability.
- Keep the exporter binary and unit root-owned. The service identity should not be able to replace its executable or systemd policy.
- Disable collectors only after understanding observability impact; enable optional collectors only with measured cost and cardinality.
- Protect Prometheus and Grafana themselves with authentication, TLS, least privilege, backups, and network policy.
Stop before continuing if
- Stop if the release checksum, architecture, source, or binary version cannot be verified.
- Stop if port 9100 binds to an unintended address or is reachable from an untrusted source.
- Stop if promtool rejects the configuration or reload changes unrelated targets.
- Stop if the target is down, samples are stale, labels are wrong, or cardinality rises unexpectedly.
- Do not call the rollout complete until a test alert reaches the intended owner and recovery/rollback is documented.
command
Create a service account
Use a non-login system identity.
Why this step matters
A non-login account gives the exporter enough identity to run while keeping interactive access, home content, and ownership of its executable outside the service boundary.
What to understand
The exporter normally reads world-readable kernel and filesystem interfaces and does not need root. Verify optional collectors before adding groups or capabilities.
--no-create-home avoids an unused writable home and nologin rejects ordinary sessions. The unit adds further process restrictions.
System changes
- Creates the persistent node_exporter system account and matching group.
Syntax explained
--system- Creates an account in the system-user range.
--no-create-home- Avoids provisioning an unnecessary writable home directory.
--shell /usr/sbin/nologin- Rejects ordinary interactive login.
sudo useradd --system --no-create-home --shell /usr/sbin/nologin node_exporternode_exporter system user created
Checkpoint: Verify the exporter identity
getent passwd node_exporter && id node_exporterContinue whenThe account uses nologin, has no unexpected groups, and owns no executable or system configuration.
Stop whenThe name already belongs to another service or has broad group membership.
If this step fails
The account already exists with a normal shell.
Likely causeA prior manual deployment or unrelated package created it differently.
getent passwd node_exporterid node_exporterfind / -xdev -user node_exporter -ls 2>/dev/null | head
ResolutionInventory the existing deployment and migrate intentionally; do not alter an unknown account in place.
Security notes
- Do not add node_exporter to privileged device or log-reader groups without a collector-specific requirement.
Alternatives
- Use the identity created by a trusted distribution package.
Stop conditions
- No shared or interactive exporter account proceeds.
command
Install a verified binary
Verify the release checksum and install it root-owned.
Why this step matters
Checksum verification binds the installed executable to the reviewed official release artifact before a root-owned path makes it part of the service supply chain.
What to understand
Verify the checksum manifest and release provenance before extracting. The example assumes node_exporter is the exact architecture-specific binary selected.
install places it root-owned mode 0755 so the service can execute but cannot replace it. Record `node_exporter --version` and file digest after installation.
System changes
- Installs or replaces /usr/local/bin/node_exporter with the verified root-owned executable.
Syntax explained
sha256sum --check- Compares the downloaded binary digest with the official manifest entry.
install -o root -g root -m 0755- Copies the verified binary with root ownership and executable permissions.
sha256sum --check sha256sums.txt && sudo install -o root -g root -m 0755 node_exporter /usr/local/bin/node_exporternode_exporter: OK
Checkpoint: Record installed provenance
sha256sum /usr/local/bin/node_exporter && /usr/local/bin/node_exporter --versionContinue whenThe digest matches the reviewed release and version/architecture are expected.
Stop whenManifest, digest, source, architecture, or version differs.
If this step fails
sha256sum reports FAILED.
Likely causeWrong architecture/archive, incomplete download, mismatched manifest, or tampering.
file node_exportersha256sum node_exportergrep 'node_exporter' sha256sums.txt
ResolutionDelete the untrusted download, reacquire release and manifest from the official project, and verify before installation.
Security notes
- Do not execute an unverified downloaded binary even just to ask for its version.
Alternatives
- Install a signed distribution package and verify repository metadata.
Stop conditions
- Checksum failure blocks installation.
config
Create the systemd unit
Bind to a management address and add compatible sandboxing.
Why this step matters
The systemd unit fixes the listener and identity while applying filesystem, home, temporary-directory, and privilege restrictions compatible with read-only host metric collection.
What to understand
The explicit management address avoids a wildcard listener. Prometheus must route to that address and the host firewall narrows the source.
ProtectSystem=strict and ProtectHome=true restrict filesystem views; PrivateTmp isolates temporary files and NoNewPrivileges blocks exec-based privilege gains.
Restart=on-failure recovers unexpected exits. Add rate limits and resource limits based on measured service behavior.
System changes
- Creates /etc/systemd/system/node_exporter.service; the service is not started yet.
Syntax explained
User=node_exporter- Runs the exporter as the dedicated unprivileged identity.
--web.listen-address- Binds HTTP metrics to the selected management address and TCP 9100.
NoNewPrivileges=true- Prevents descendants from gaining privilege through exec.
ProtectSystem=strict- Makes broad filesystem paths read-only inside the service mount namespace.
WantedBy=multi-user.target- Defines normal boot enablement.
/etc/systemd/system/node_exporter.serviceValues stay on this page and are never sent or saved.
[Unit]
Description=Prometheus Node Exporter
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=60s
StartLimitBurst=5
[Service]
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter --web.listen-address={{managementAddress}}:9100
Restart=on-failure
RestartSec=5s
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectControlGroups=true
[Install]
WantedBy=multi-user.targetUnit saved
Checkpoint: Validate the unit before loading
sudo systemd-analyze verify /etc/systemd/system/node_exporter.serviceContinue whenNo output and exit code zero; effective listener contains the reviewed management address.
Stop whenAny directive is unsupported, binary is missing, or listener is wildcard/public.
If this step fails
The unit fails verification because the binary path does not exist.
Likely causeThe release was installed elsewhere or the architecture extraction path was copied incorrectly.
namei -l /usr/local/bin/node_exporterfile /usr/local/bin/node_exporter
ResolutionInstall the verified binary at the declared path or update the unit deliberately and repeat validation.
Security notes
- Keep the unit root-owned and do not place credentials in ExecStart.
Alternatives
- Use a distribution package unit after reviewing its effective flags and sandbox.
Stop conditions
- Do not daemon-reload an invalid unit.
command
Start and inspect
Start the unit and verify the exact listener.
Why this step matters
Starting, enabling, checking unit state, reading logs, and inspecting the exact socket together prove the intended revision is running on only the management interface.
What to understand
daemon-reload reads the new unit; enable --now creates boot links and starts it. Record these state changes separately in production.
ss must show managementAddress:9100 owned by node_exporter rather than 0.0.0.0, [::], or another process.
Review journal startup lines for enabled collectors, filesystem exclusions, permission errors, and version.
System changes
- Reloads systemd, enables Node Exporter at boot, starts the process, and opens the selected local listener.
Syntax explained
daemon-reload- Reloads unit definitions into the systemd manager.
enable --now- Enables at boot and starts immediately.
ss -lntp- Shows listening TCP sockets and owning processes.
sudo systemctl daemon-reload && sudo systemctl enable --now node_exporter && ss -lntp | grep ':9100'LISTEN ... 10.0.10.15:9100
Checkpoint: Confirm the exact listener
systemctl is-active node_exporter && ss -lntp 'sport = :9100' && journalctl -u node_exporter -n 50 --no-pagerContinue whenService is active with one listener on 10.0.10.15:9100 and no collector errors.
Stop whenThe listener is wildcard/public, process differs, logs show errors, or restart count rises.
If this step fails
Node Exporter cannot bind the management address.
Likely causeThe address is not configured locally, another process owns 9100, or the unit has a typo.
ip -brief addressss -lntp 'sport = :9100'journalctl -u node_exporter -n 100 --no-pager
ResolutionCorrect the stable local address or port plan; do not fall back to a wildcard listener.
Security notes
- A private-address listener still needs source-restricted firewall policy.
Alternatives
- Bind loopback and scrape through an authenticated local proxy or agent.
Stop conditions
- Do not create a Prometheus target until local listener identity is correct.
warning
Restrict metrics access
Metrics expose host details; permit port 9100 only from Prometheus.
Why this step matters
Host metrics expose infrastructure detail, so one source-restricted rule must be merged into the existing persistent policy with console recovery and syntax validation.
What to understand
Match both destination TCP port 9100 and the exact Prometheus source address. Place the rule before a default drop without broadening other traffic.
Validate the complete candidate with nft --check and keep current management access plus a saved ruleset. Test permitted and denied source paths after apply.
The displayed command validates only; applying the complete policy remains a controlled firewall change with rollback.
System changes
- The approved follow-up changes the persistent firewall to allow Prometheus and deny untrusted sources to TCP 9100.
Syntax explained
nft --check- Parses and validates the full ruleset without applying it.
--file /etc/nftables.conf- Checks the exact persistent candidate file.
ip saddr {{prometheusAddress}} tcp dport 9100 accept- Represents the narrow intended rule to merge into the existing input chain.
sudo nft --check --file /etc/nftables.confFirewall syntax valid with source-restricted TCP 9100
Checkpoint: Prove both network paths
curl --fail http://10.0.10.15:9100/metrics | headContinue whenPrometheus source succeeds; an independent untrusted source times out or is rejected.
Stop whenSyntax fails, current SSH is at risk, untrusted source succeeds, or Prometheus cannot connect.
If this step fails
The exporter is reachable from the internet despite an nftables rule.
Likely causeTraffic uses another interface/address family, an earlier accept wins, or cloud/network firewall policy differs.
sudo nft -a list rulesetss -lntp 'sport = :9100'ip -brief address
ResolutionRemove unintended exposure at listener and every filtering layer, then retest externally before continuing.
Security notes
- Keep console or a second session during firewall changes.
Alternatives
- Use mTLS through a reverse proxy when source IP restriction is insufficient.
Stop conditions
- Dangerous firewall application requires explicit rollback and an external negative test.
config
Add the Prometheus target
Use stable labels and validate Prometheus before reload.
Why this step matters
Prometheus needs an explicit stable target and low-cardinality labels, and the entire merged configuration must pass promtool before a reload can affect existing monitoring.
What to understand
job_name groups the exporter family. The static target names address:port and environment is a stable operational label.
Do not add hostnames, request IDs, or other unbounded values as target labels. Use service discovery for fleet churn rather than manual edits at scale.
Run promtool check config on the full file, review the diff, then reload Prometheus and inspect target health.
System changes
- Adds one scrape target and stable label to Prometheus configuration; the approved reload changes future collection.
Syntax explained
job_name: node- Assigns the stable job label used by dashboards and rules.
targets- Lists the exact management endpoint Prometheus scrapes.
environment- Adds one controlled low-cardinality context label.
/etc/prometheus/prometheus.ymlValues stay on this page and are never sent or saved.
scrape_configs:
- job_name: node
static_configs:
- targets: ['{{managementAddress}}:9100']
labels:
environment: '{{environmentLabel}}'SUCCESS: configuration is valid
Checkpoint: Validate before reload
promtool check config /etc/prometheus/prometheus.ymlContinue whenSUCCESS for the complete configuration and no unrelated target diff.
Stop whenValidation fails, duplicate labels collide, or target points to an unintended address.
If this step fails
Promtool reports duplicate scrape config or YAML errors.
Likely causeThe snippet was pasted at the wrong indentation or duplicates an existing job_name.
promtool check config /etc/prometheus/prometheus.ymlgrep -n 'job_name:' /etc/prometheus/prometheus.yml
ResolutionCorrect the merged YAML and use a unique reviewed job before reload.
Security notes
- Prometheus configuration can contain credentials for other targets; restrict file and review output.
Alternatives
- Use file-based discovery generated by configuration management.
Stop conditions
- Never reload invalid or unreviewed Prometheus configuration.
verification
Verify the scrape
Confirm the target returns up=1 before configuring Grafana.
Why this step matters
The API query verifies that Prometheus, not merely a local curl, successfully scraped and stored a fresh sample under the intended labels.
What to understand
up is generated per scrape target: 1 means the latest scrape succeeded and 0 means it failed. It does not mean host workloads are healthy.
Inspect /api/v1/targets for last scrape time, duration, labels, and error. Query node_uname_info and a filesystem metric to prove real exporter content.
Check sample freshness and clock synchronization so a stale cached value cannot satisfy acceptance.
System changes
- No change; queries the Prometheus HTTP API for stored target health.
Syntax explained
/api/v1/query- Executes an instant PromQL query through Prometheus.
up{job="node"}- Selects current scrape success for the node job.
%7B...%7D- URL-encodes PromQL braces and quotes for the HTTP request.
curl --fail 'http://prometheus:9090/api/v1/query?query=up%7Bjob%3D%22node%22%7D'{"status":"success","data":{"result":[{"value":[...,"1"]}]}}Checkpoint: Require fresh node samples
promtool query instant http://prometheus:9090 'up{job="node",environment="production"}'Continue whenExactly the intended instance returns 1 and its last scrape is recent.
Stop whenNo result, multiple unexpected instances, value 0, wrong labels, stale time, or scrape error.
If this step fails
The query returns multiple nodes with indistinguishable instance labels.
Likely causeRelabeling overwrote unique instance identity or targets share an incorrect label.
promtool query instant http://prometheus:9090 'count by (job,instance,environment) (up{job="node"})'
ResolutionRestore stable unique instance labels before building dashboards and alerts.
Security notes
- Protect Prometheus API access because queries expose infrastructure inventory.
Alternatives
- Inspect the Prometheus Targets UI from the protected management network.
Stop conditions
- Do not configure dashboards around a target that is not fresh and uniquely labeled.
instruction
Connect Grafana
Add Prometheus as a data source and build or import a reviewed dashboard.
Why this step matters
A reviewed dashboard turns raw series into operational context and must prove its data source, variables, units, time range, and freshness rather than merely rendering panels.
What to understand
Configure Prometheus with a protected internal URL and Save & test. Use least-privilege Grafana roles and version dashboard JSON.
Start with CPU saturation, memory pressure, disk bytes and inodes, I/O, network errors, load, uptime, and scrape health. Display units and rates correctly.
Review imported dashboards for unexpected data sources, variables, hidden queries, high-cardinality expressions, and unsupported metric names.
System changes
- Creates or updates a Grafana Prometheus data source and versioned host dashboard.
Syntax explained
Data source URL- Server-side Grafana connection to the protected Prometheus endpoint.
Dashboard variable- Controlled label selector for job, instance, and environment.
Panel query- PromQL expression with explicit units, legend, range, and resolution.
Current host metrics appear with the expected node labels
Checkpoint: Validate dashboard truth
Continue whenPanels show the selected host, current timestamp, expected units, and values that agree with direct Prometheus queries.
Stop whenA panel has no data, wrong instance, stale sample, unexplained unit, or expensive query.
If this step fails
Dashboard panels show N/A after import.
Likely causeDashboard variable, job label, metric version, or data source UID differs.
promtool query instant http://prometheus:9090 'node_uname_info{job="node"}'curl --fail http://grafana:3000/api/health
ResolutionMap variables and queries to the verified local labels instead of enabling random collectors.
Security notes
- Do not make operational dashboards anonymously public.
Alternatives
- Build a minimal internal dashboard rather than importing a broad community dashboard.
Stop conditions
- No dashboard is accepted without direct-query comparison.
instruction
Add actionable alerts
Cover host down, disk, inode, memory, and sustained CPU with owners and runbooks.
Why this step matters
Monitoring only reduces incident time when important conditions reach an accountable person with context, urgency, and a tested diagnostic path.
What to understand
Cover exporter down, disk byte and inode exhaustion, sustained CPU saturation, memory pressure, and scrape failure using duration windows that avoid transient noise.
Every alert needs severity, owner/team, summary, description, dashboard, and runbook labels or annotations. Route and inhibition rules must be reviewed.
Use a temporary controlled test rule or short exporter stop to exercise evaluation, routing, delivery, acknowledgement, and resolution.
System changes
- Adds alert rules and notification routing; the test generates one deliberate firing and resolved event.
Syntax explained
expr- PromQL condition defining the monitored risk.
for- Requires the condition to persist before firing.
labels- Carries stable severity and ownership routing context.
annotations- Supplies operator-facing summary, evidence, dashboard, and runbook.
A test alert reaches its notification route
Checkpoint: Exercise the complete alert path
Continue whenThe controlled alert becomes pending, fires, reaches the intended receiver with runbook context, and resolves after recovery.
Stop whenRule validation fails, alert routes to the wrong audience, notification lacks context, or resolution is not observed.
If this step fails
The alert remains pending and never fires during the test.
Likely causeThe expression, labels, evaluation interval, for duration, or test signal does not match.
promtool check rules /etc/prometheus/rules/node.ymlcurl --fail http://prometheus:9090/api/v1/rulescurl --fail http://prometheus:9090/api/v1/alerts
ResolutionInspect rule state and exact series labels, correct the test safely, and restore production durations after validation.
Security notes
- Notification payloads must not leak secrets or sensitive host details to broad channels.
Alternatives
- Test routing with a dedicated always-firing canary rule clearly labeled as a drill.
Stop conditions
- Do not enable noisy or ownerless alerts.
Finish line
Verification checklist
curl --fail http://10.0.10.15:9100/metrics | headReturns Prometheus metrics.promtool query instant http://prometheus:9090 'up{job="node"}'Returns 1.Open the host dashboardPanels show fresh samples.Recovery guidance
Common problems and safe checks
Prometheus reports the node target down while local curl works.
Likely causeFirewall source policy, routing, listener address, target address, or Prometheus host identity differs from the plan.
ss -lntp 'sport = :9100'sudo nft list rulesetcurl --fail http://10.0.10.15:9100/metrics | headcurl --fail http://prometheus:9090/api/v1/targets
ResolutionCorrect the narrow listener, route, target, or source-restricted rule; do not open 9100 broadly as a diagnostic shortcut.
Node Exporter starts but logs collector permission errors.
Likely causeA sandbox directive or host mount prevents required read access, or an optional collector expects privileged data.
journalctl -u node_exporter -n 100 --no-pagersystemctl show node_exporter -p ProtectSystem -p ProtectHome -p Usercurl -s http://10.0.10.15:9100/metrics | grep '^node_scrape_collector_'
ResolutionDocument the collector requirement and add the narrowest compatible read access or disable that optional collector with stated monitoring loss.
Grafana shows no data although Prometheus reports up=1.
Likely causeThe data source URL, time range, dashboard labels, instance selector, or clock differs from the stored series.
curl --fail 'http://prometheus:9090/api/v1/query?query=node_uname_info'timedatectl statuscurl --fail http://grafana:3000/api/health
ResolutionCorrect the data source and dashboard variables, then query a known series directly before importing more panels.
A host-down test fires in Prometheus but no notification arrives.
Likely causeAlert routing, grouping, inhibition, receiver credentials, or ownership labels are incomplete.
promtool check rules /etc/prometheus/rules/*.ymlcurl --fail http://prometheus:9090/api/v1/alertscurl --fail http://alertmanager:9093/api/v2/status
ResolutionCorrect and test the notification route with a temporary clearly labeled alert before relying on it for incidents.
Reference
Frequently asked questions
Why not bind Node Exporter to 127.0.0.1?
That is ideal when a local Prometheus agent or protected proxy performs collection. A remote Prometheus needs a routable management address, restricted by firewall or authenticated transport.
Does up=1 mean the host is healthy?
It means Prometheus successfully scraped the target. CPU, memory, disks, services, and applications can still be unhealthy.
Should every available collector be enabled?
No. Start with defaults, understand each optional collector's permissions, cost, cardinality, and operational value, then add deliberately.
Recovery
Rollback
Remove the scrape target before stopping the exporter.
- Remove and validate the Prometheus target.
- Close TCP 9100 in the firewall.
- Disable and stop node_exporter.
- Remove binary and unit after dashboards no longer depend on them.
Evidence