Diagnose packet loss, MTU, routing, and DNS failures on Linux
Turn a vague report such as “the network is slow” into timestamped evidence by separating name resolution, route selection, neighbor reachability, physical counters, latency/loss, Path MTU, transport establishment, and application response—before changing any state.
Identify the failing network layer and hand the responsible operator a minimal, privacy-conscious evidence bundle, using bounded probes and narrowly filtered packet captures only after read-only host observations have eliminated simpler explanations.
- Ubuntu Server 24.04 LTS
- iproute2 6.1
- tcpdump 4.99
- traceroute 2.1
- BIND utilities 9.18
- Precise symptom and consent Record affected client, destination, protocol, start/end time, expected behavior, and whether the problem is continuous or intermittent. Obtain authorization for active probes and packet metadata capture.
- Known comparison point Use one healthy client, alternate destination, or previous baseline from the same network. Without comparison, a single latency or route observation is easy to misinterpret.
- Clock and scope Confirm source and destination clocks and timezone. Define {{destinationHost}}, {{destinationIp}}, {{destinationPort}}, {{interfaceName}}, and a maximum capture duration/packet count before running tools.
date --iso-8601=seconds; timedatectl show -p NTPSynchronized --value - Privacy handling Choose a protected evidence directory, restrict it to the operator, and define who may receive addresses, hostnames, DNS queries, packet headers, and timing data. Never capture unrelated interfaces or full payload by default.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A layer-by-layer incident record that separates local host load, interface health, route/source selection, neighbor state, DNS, path behavior, PMTU, transport, and application response.
- A reproducible bounded probe contract using the same destination, source, protocol, time window, and healthy comparison.
- A minimal packet-control-plane capture with explicit duration, count, snap length, filter, permissions, retention, and redaction rules.
- An evidence-based ownership decision that tells the host, LAN, DNS, transit, firewall, remote service, or application team what to inspect next.
- The symptom can be reproduced or bounded with exact timestamps and a safe transaction rather than described only as intermittent slowness.
- The kernel-selected route, source, next hop, neighbor, DNS answer, link counters, path latency, and PMTU are recorded without changing network state.
- If capture is needed, it contains only control packets for one authorized host/port and can be explained packet by packet.
- The diagnosis is supported by at least two independent observations and a healthy comparison, and the original test proves recovery after the owning team fixes the cause.
Architecture
How the parts fit together
The workflow treats a network request as a sequence of dependencies: local process and resource readiness, name resolution, policy route and source selection, neighbor resolution on the first link, physical/queue delivery, routed path and PMTU, remote transport response, and application/TLS response. Read-only tools observe each boundary in that order. Only after the failure boundary is narrow does tcpdump collect limited control-plane metadata. A stable comparison path separates destination-specific failure from source-wide failure.
- The client resolves {{destinationHost}} through {{resolverIp}} and selects an address according to application/family policy.
- The kernel applies policy rules, selects a route table, source {{localSourceIp}}, interface {{interfaceName}}, and next-hop neighbor.
- The NIC and qdisc transmit frames; counters reveal local corruption, buffering, or queue drops.
- Routers forward packets subject to ACL, ECMP, congestion, and path MTU; ICMP feedback and trace tools expose some of that state.
- The destination or intervening firewall returns SYN-ACK, RST, ICMP error, or nothing; socket state and capture identify direction.
- After transport/TLS, application timing separates DNS, connect, TLS, first-byte, and total delay.
- The operator assigns ownership to the narrowest evidenced layer and reruns the original contract after remediation.
Assumptions
- The operator is authorized to send low-rate ICMP, UDP-style trace, TCP SYN, DNS, and HEAD health requests to the named systems.
- {{destinationIp}} and {{destinationIPv6}} are approved literal addresses corresponding to the incident; any load balancer rotation is recorded.
- The host clock is synchronized closely enough to correlate with remote, firewall, and packet timestamps.
- Packet capture is a last-mile diagnostic with protected handling. Full payload, broad subnets, `-i any`, unlimited duration, and unrelated DNS are outside the default scope.
- Intermediate traceroute silence is not automatically loss. Routers may rate-limit or filter probes while forwarding production traffic normally.
- A single source cannot prove symmetric routing. When necessary, a remote operator captures the same transaction or supplies firewall/load-balancer logs.
- This tutorial diagnoses before remediation. Route, MTU, DNS, firewall, sysctl, and service changes require a separate reviewed change.
Key concepts
- Packet loss
- Packets sent but not delivered or acknowledged. Loss may occur at local queues, physical media, congestion, policy devices, remote queues, or because the responder intentionally rate-limits probes.
- Policy routing
- Rules select route tables based on source, destination, marks, or other fields. The main table alone may not describe the route an application uses.
- Neighbor reachability
- ARP for IPv4 and ND for IPv6 map an on-link next hop to link-layer identity and track reachability states.
- Path MTU
- The smallest link MTU on one direction of one path. Incorrectly filtered ICMP can create black holes where small packets work and larger transfers stall.
- ECMP
- Equal-cost multipath may hash flows onto different paths. Repeated trace or port changes can legitimately show different hops and loss characteristics.
- Snap length
- The number of bytes retained from each captured packet. A limited snap length can preserve headers while minimizing application payload collection.
- BPF capture filter
- Kernel-side expression that selects packets before user-space capture, reducing volume and privacy exposure.
- SYN retransmission
- Repeated TCP SYN without SYN-ACK or RST indicates no transport response reached the source, but not by itself which intermediate device dropped it.
Before you copy
Values used in this guide
{{interfaceName}}Kernel interface selected by the route to the failing destination.
Example: ens3{{localSourceIp}}Expected source address chosen for the failing transaction.
Example: 198.51.100.20{{destinationHost}}Exact application hostname from the incident.
Example: api.example.net{{destinationIp}}Approved IPv4 address selected for literal and capture tests.
Example: 203.0.113.80{{destinationIPv6}}Approved IPv6 address for family-specific PMTU comparison.
Example: 2001:db8:500::80{{destinationPort}}Transport port used by the failing service.
Example: 443{{resolverIp}}Configured recursive resolver queried directly for evidence.
Example: 192.0.2.53{{gatewayIp}}First-hop gateway selected for the destination route.
Example: 198.51.100.1{{comparisonIp}}Stable approved target used to compare source-wide versus destination-specific loss.
Example: 192.0.2.80{{probeCount}}Small common count for comparable gateway, destination, and control probes.
Example: 20{{captureSeconds}}Maximum duration of the scoped packet capture.
Example: 30{{capturePackets}}Maximum accepted packets before tcpdump stops.
Example: 100Security and production boundaries
- Packet metadata can reveal internal addresses, hostnames, user timing, topology, sequence numbers, DNS queries, and quoted payload fragments in ICMP errors. Handle it as sensitive incident data.
- Default to numeric output (`-n`/`-nn`) so tools do not generate extra DNS traffic or expose unrelated query names during diagnosis.
- Never capture credentials or decrypted application payload. Limit snap length, filter, interface, host, port, packet types, duration, and count before starting.
- Do not upload raw pcap to public tickets or AI services. Produce a redacted textual timeline and retain the original only in approved encrypted evidence storage.
- Low-rate probes are still active network traffic. Respect provider terms, change windows, and fragile devices; stop when they affect service.
- A route or DNS flush can temporarily hide evidence and disrupt unrelated applications. This guide intentionally avoids such state changes.
Stop before continuing if
- Stop active probes if they increase loss, trigger security controls, affect a fragile service, or exceed authorization.
- Do not capture until the destination, interface, filter, snap length, duration, count, storage, and recipient are explicitly approved.
- Stop if the route chooses a different interface/source than the capture plan; correct scope before collecting packets.
- Do not diagnose intermediate traceroute silence as loss without end-to-end evidence and a comparison.
- Do not change MTU, route, neighbor, DNS, firewall, offload, or service state inside this diagnostic workflow.
- Stop escalation claims when timestamps, clock, resolver answer, or load-balancer destination cannot be correlated.
instruction
Frame one reproducible failing transaction
Write a short test contract that names the client, hostname and literal address, protocol/port, expected status, observed error, time window, frequency, and a safe health request. Reproduce once without changing DNS, routes, MTU, firewall, or services.
Why this step matters
A single safe, reproducible transaction prevents different operators from testing different addresses, ports, methods, or time windows and then drawing incompatible conclusions.
What to understand
Record exact error and timestamps, not a paraphrase. A timeout, refusal, TLS failure, SERVFAIL, and HTTP 503 identify different layers.
HEAD to a documented health path avoids state-changing application actions; confirm the endpoint is safe before use.
Keep hostname and literal address because load balancers and DNS rotation can change the actual target between attempts.
System changes
- No local persistent change; sends one bounded, safe application health request.
Syntax explained
--head- Requests headers only from a health endpoint without a response body.
--connect-timeout 3- Bounds DNS-complete transport establishment.
--max-time 8- Bounds the complete request.
--fail --show-error- Returns nonzero for HTTP errors while preserving a concise error.
Values stay on this page and are never sent or saved.
date --iso-8601=seconds; printf 'source=%s destination=%s address=%s port=%s\n' "$(hostname -f)" '{{destinationHost}}' '{{destinationIp}}' '{{destinationPort}}'; curl --head --fail --silent --show-error --connect-timeout 3 --max-time 8 https://{{destinationHost}}/health2026-07-28T11:20:14+00:00 source=app01.example.net destination=api.example.net address=203.0.113.80 port=443 curl: (28) Connection timed out after 3001 milliseconds
Checkpoint: Freeze the test contract
Continue whenSource, hostname, address, port, method, expected response, exact error, timestamps, and limits are recorded.
Stop whenThe request can mutate data, target rotates without recording, or authorization is unclear.
If this step fails
The failure cannot be reproduced.
Likely causeIt is intermittent, client-specific, cached, load-balanced, or already recovered.
date --iso-8601=secondsgetent ahosts api.example.netcurl --head --max-time 8 https://api.example.net/health
ResolutionDo not generate load. Define a low-rate observation window and capture comparison telemetry until one failing transaction is timestamped.
Security notes
- Never place tokens, cookies, or customer identifiers in the diagnostic command.
Alternatives
- Use an application-provided no-side-effect readiness endpoint or synthetic transaction.
Stop conditions
- No diagnosis without a stable target and safe method.
verification
Capture host, interface, socket, and resource baseline
Record uptime, load, failed services, interface/address state, socket summary, conntrack pressure indicators, and kernel network warnings. A saturated host or failed local service can look exactly like packet loss.
Why this step matters
Local CPU, service, socket, conntrack, or kernel failure can create timeouts and loss before any packet reaches the wire. Establishing host health prevents premature network escalation.
What to understand
Load average needs CPU/run-queue context, but a normal lightweight baseline can still expose an obvious host-wide incident.
Conntrack near maximum can drop new flows while existing connections continue, closely resembling firewall or remote failure.
Kernel messages reveal carrier transitions, watchdog timeouts, queue resets, neighbor failures, and table exhaustion at the incident time.
System changes
- No persistent change; reads host, service, socket, sysctl, and kernel-log state.
Syntax explained
ss --summary- Summarizes socket states without enumerating payload or application data.
nf_conntrack_count/max- Compares current tracked flows with the table ceiling.
journalctl -k --since- Reads bounded recent kernel messages.
date --iso-8601=seconds; uptime; systemctl --failed --no-pager; ip -brief link; ip -brief address; ss --summary; sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max 2>/dev/null; journalctl -k --since '-30 minutes' --no-pager | grep -iE 'link.*(down|up)|NETDEV WATCHDOG|tx timeout|martian|conntrack|neigh.*fail' || true2026-07-28T11:21:02+00:00 11:21:02 up 37 days, 3:42, 2 users, load average: 0.18, 0.21, 0.20 0 loaded units listed. ens3 UP 198.51.100.20/24 2001:db8:120::20/64 Total: 211 TCP: 84 (estab 23, closed 45, orphaned 0, timewait 16) net.netfilter.nf_conntrack_count = 418 net.netfilter.nf_conntrack_max = 262144
Checkpoint: Rule out host-wide pressure
Continue whenNo failed critical service, saturation, conntrack exhaustion, watchdog, or link event overlaps the symptom.
Stop whenA local host incident already explains the failure and needs its own response.
If this step fails
Conntrack count is within 2% of maximum.
Likely causeConnection surge, leak, attack, too-small table, or long timeouts.
sudo conntrack -S 2>/dev/nullss --summaryjournalctl -k | grep -i conntrack
ResolutionEscalate as host/firewall capacity with flow-rate evidence; do not flush conntrack because it disrupts every tracked connection.
Security notes
- Socket counts are lower sensitivity than full process/peer lists; keep collection minimal.
Alternatives
- Use approved node telemetry covering the exact incident window.
Stop conditions
- Stop path probing if the host is already unstable.
verification
Inspect carrier, driver, MTU, queue, and error counters
Read the selected interface's negotiated state, MTU, qdisc statistics, kernel counters, and driver-specific errors twice around a short interval. Counter deltas are more useful than a large lifetime total.
Why this step matters
Physical corruption, carrier flaps, driver resets, buffer misses, and qdisc drops are observable at the source and can localize loss before routed-path tools introduce ambiguity.
What to understand
Capture counters twice around the failing request and compute deltas; lifetime totals from months of uptime are not incident evidence.
Driver counters are vendor-specific. Preserve names and values with driver/firmware version rather than assuming every `drop` means congestion.
qdisc backlog/drops indicate local queueing; NIC CRC or carrier errors point lower in the stack.
System changes
- No persistent change; reads link, driver, and qdisc counters.
Syntax explained
ip -statistics link- Shows standard packet/error/drop counters and current MTU/qdisc.
ethtool -S- Shows driver-specific hardware and queue counters.
tc -s qdisc- Shows scheduler backlog, drops, overlimits, and timing.
Values stay on this page and are never sent or saved.
ip -details -statistics link show {{interfaceName}}; sudo ethtool {{interfaceName}}; sudo ethtool -S {{interfaceName}} | grep -iE 'crc|error|drop|miss|timeout|reset|no.?buffer' | head -n 40; tc -s qdisc show dev {{interfaceName}}2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP
RX: bytes packets errors dropped missed mcast
928412049 1149021 0 0 0 1821
TX: bytes packets errors dropped carrier collsns
701184201 901442 0 0 0 0
Speed: 10000Mb/s
Duplex: Full
Link detected: yes
qdisc fq_codel 0: root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100msCheckpoint: Classify local link evidence
Continue whenCarrier/speed are stable and counter deltas show either no local loss or a specific reproducible error/drop class.
Stop whenCarrier flaps or error counters rise fast enough that further probes could worsen service.
If this step fails
TX dropped rises while NIC errors remain zero.
Likely causeLocal qdisc congestion, shaping, queue limit, or application burst rather than physical media.
tc -s qdisc show dev ens3ip -s link show ens3sar -n DEV 1 5 2>/dev/null
ResolutionEscalate with queue and traffic-rate evidence to capacity/QoS ownership; do not enlarge queues blindly and add latency.
Security notes
- Counter output is normally safe but includes interface names and capacity.
Alternatives
- Use switch port and NIC telemetry for the same timestamps.
Stop conditions
- Physical instability must be stabilized before route/path conclusions.
verification
Resolve policy routing, source selection, and next-hop neighbor
Ask the kernel for the exact route to the literal destination from the intended source, then inspect rules, the selected table, next hop, and neighbor state. This catches wrong source addresses, stale gateways, asymmetric policy tables, and incomplete ARP/ND before path probes.
Why this step matters
The kernel's actual source, rule, table, next hop, and neighbor determine where packets go. Reading only the main route table can miss the policy that affects the failing application.
What to understand
`ip route get` simulates one lookup with explicit source and reveals table, gateway, interface, and selected source.
Neighbor FAILED/INCOMPLETE localizes failure to first-hop resolution, VLAN, or gateway reachability rather than remote transit.
`ip monitor` records state changes during reproduction without modifying caches.
System changes
- No persistent change; reads policy routes and monitors ten link/route/neighbor events.
Syntax explained
ip rule show- Lists policy-routing rule precedence and table selection.
ip route get destination from source- Resolves the exact route for the intended flow source.
ip neigh show- Displays ARP/ND link-layer resolution and reachability.
ip monitor ... count 10- Stops after ten kernel routing-related events.
Values stay on this page and are never sent or saved.
ip rule show; ip route get {{destinationIp}} from {{localSourceIp}}; ip route show table all; ip neigh show dev {{interfaceName}}; ip monitor neigh route link count 100: from all lookup local
100: from 198.51.100.20 lookup service
32766: from all lookup main
203.0.113.80 via 198.51.100.1 dev ens3 table service src 198.51.100.20 uid 1000
cache
198.51.100.1 dev ens3 lladdr 00:11:22:33:44:55 REACHABLECheckpoint: Approve first-hop path
Continue whenIntended source, table, interface, next hop, and reachable neighbor are consistent with the network design.
Stop whenRoute uses an unexpected source/table/interface or neighbor remains failed.
If this step fails
Route-get chooses the right gateway but wrong source address.
Likely causeMultiple addresses, policy rules, route `src`, application bind, or address deprecation changes source selection.
ip address show dev ens3ip rule showip route show table all
ResolutionIdentify the policy owner and reproduce with the application's actual bind/source. Change routing only in a separate reviewed remediation.
Security notes
- Policy tables and gateways expose internal topology; redact before external sharing.
Alternatives
- Use `networkctl status` and application socket binding as corroborating evidence.
Stop conditions
- Do not capture on an interface different from route-get output.
verification
Separate DNS data, resolver transport, and application caching
Query the configured resolver and an approved comparison resolver for A, AAAA, CNAME, TTL, status, and timing; query the literal IP separately at the application layer. Do not change `/etc/resolv.conf` or flush caches until the evidence explains a DNS-specific failure.
Why this step matters
DNS failure may be wrong data, resolver transport, split horizon, family selection, TTL/cache, or application caching. Direct and system queries separate those layers without changing resolver state.
What to understand
`resolvectl` shows which link and protocol supplied the system answer; `dig @resolver` tests one recursive server directly.
A and AAAA must both be recorded because clients may prefer a failing IPv6 address even when IPv4 is healthy.
TTL and query time distinguish stale/rotating data from slow resolver transport. A literal-IP application test still needs correct Host/SNI.
System changes
- No persistent change; sends bounded DNS queries to the configured resolver.
Syntax explained
dig @{{resolverIp}}- Queries the selected recursive resolver directly.
+noall +answer +comments +stats- Limits output to status, answers, timing, and server metadata.
getent ahosts- Shows addresses returned through the system name-service stack used by many applications.
Values stay on this page and are never sent or saved.
resolvectl status --no-pager; resolvectl query {{destinationHost}}; dig @{{resolverIp}} {{destinationHost}} A +noall +answer +comments +stats; dig @{{resolverIp}} {{destinationHost}} AAAA +noall +answer +comments +stats; getent ahosts {{destinationHost}}api.example.net: 203.0.113.80 -- link: ens3
2001:db8:500::80 -- link: ens3
-- Information acquired via protocol DNS in 18.4ms.
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38140
api.example.net. 60 IN A 203.0.113.80
;; Query time: 17 msec
;; SERVER: 192.0.2.53#53(192.0.2.53) (UDP)Checkpoint: Classify DNS independently
Continue whenSystem and direct resolver answers, families, TTLs, status, and timing are understood and compared with authoritative expectation.
Stop whenSERVFAIL, stale/unexpected address, split answer, or resolver timeout is unexplained.
If this step fails
Direct dig works but the application resolves differently.
Likely causesystemd-resolved link routing, NSS order, local hosts file, application cache, proxy, or DoH bypass.
resolvectl query api.example.netgetent ahosts api.example.netgrep '^hosts:' /etc/nsswitch.confgrep -n 'api.example.net' /etc/hosts
ResolutionTrace the application's resolver path and cache; do not overwrite `/etc/resolv.conf` as a shortcut.
Security notes
- Query only the incident hostname; DNS can expose user behavior and internal service names.
Alternatives
- Use approved resolver logs with query identifiers/timestamps when client queries cannot be sent.
Stop conditions
- No network-path conclusion while destination address is unstable.
verification
Measure bounded loss and latency to local and remote landmarks
Probe the gateway, destination, and a stable comparison target with the same small count and interval. Record sent/received/loss plus min/average/max/deviation, but do not treat an intermediate router's missing echo response as proof it drops forwarded traffic.
Why this step matters
A common bounded sample to gateway, destination, and comparison distinguishes local-segment failure, destination-specific impairment, and source-wide congestion more reliably than one continuous ping.
What to understand
Use the same count and interval for comparable statistics and record exact time.
Gateway response can be rate-limited, so combine it with counters and neighbor state; destination loss is still only one protocol's response behavior.
Average hides spikes; preserve min/max/deviation and individual sequence gaps.
System changes
- No persistent change; sends {{probeCount}} low-rate ICMP echo requests to each of three approved targets.
Syntax explained
-n- Disables reverse DNS and avoids extra network queries.
-c {{probeCount}}- Bounds each sample to the same count.
-i 0.2- Spaces probes at five per second, below flood behavior.
Values stay on this page and are never sent or saved.
ping -n -c {{probeCount}} -i 0.2 {{gatewayIp}}; ping -n -c {{probeCount}} -i 0.2 {{destinationIp}}; ping -n -c {{probeCount}} -i 0.2 {{comparisonIp}}20 packets transmitted, 20 received, 0% packet loss, time 3811ms rtt min/avg/max/mdev = 0.287/0.421/0.810/0.103 ms 20 packets transmitted, 17 received, 15% packet loss, time 3824ms rtt min/avg/max/mdev = 10.841/18.320/61.418/11.201 ms 20 packets transmitted, 20 received, 0% packet loss, time 3810ms rtt min/avg/max/mdev = 9.902/10.214/10.991/0.219 ms
Checkpoint: Compare loss domains
Continue whenGateway, destination, and comparison results identify whether loss is local, destination-specific, or shared, with timestamps and variance.
Stop whenProbe rate affects service or ICMP policy makes results nonrepresentative.
If this step fails
Only the destination drops echo but TCP health is normal.
Likely causeDestination or firewall rate-limits ICMP rather than production traffic.
traceroute -n -T -p 443 203.0.113.80curl --head --max-time 8 https://api.example.net/health
ResolutionDocument ICMP policy and rely on transport/application evidence; do not report packet loss affecting the service.
Security notes
- Respect authorized rates and targets; never use flood mode.
Alternatives
- Use an approved synthetic TCP probe when ICMP is intentionally blocked.
Stop conditions
- Stop if probes trigger rate or abuse controls.
verification
Compare UDP, ICMP, and TCP path observations
Run numeric tracepath for PMTU and conventional traceroute using ICMP plus the real TCP destination port. Compare the routes and last responding hop; different results often reflect filtering or ECMP rather than a physically different failure.
Why this step matters
Routers and firewalls can treat UDP, ICMP, and TCP probes differently. Comparing them prevents a single traceroute method from being mistaken for the production path or failure.
What to understand
Numeric output avoids reverse-DNS latency/noise. Small query count and timeout bound active traffic.
TCP traceroute to the real service port better resembles firewall policy, but ECMP hashing can still change the path.
Intermediate asterisks are missing replies, not proof of forwarding loss, when later hops and destination respond.
System changes
- No persistent change; sends bounded TTL-limited UDP-style, ICMP, and TCP probes.
Syntax explained
tracepath -n- Reports hop progression and PMTU without DNS.
traceroute -I- Uses ICMP Echo probes.
traceroute -T -p port- Uses TCP SYN probes to the application port.
-q 2 -w 2- Uses two probes per hop and two-second response wait.
Values stay on this page and are never sent or saved.
tracepath -n {{destinationIp}}; traceroute -n -I -q 2 -w 2 {{destinationIp}}; traceroute -n -T -p {{destinationPort}} -q 2 -w 2 {{destinationIp}} 1?: [LOCALHOST] pmtu 1500
1: 198.51.100.1 0.420ms
2: 192.0.2.9 2.194ms
3: 203.0.113.80 18.201ms reached
Resume: pmtu 1500 hops 3 back 3
traceroute to 203.0.113.80, 30 hops max
1 198.51.100.1 0.392 ms 0.411 ms
2 * *
3 203.0.113.80 18.101 ms 18.230 msCheckpoint: Interpret path, not asterisks
Continue whenProtocol differences, last responding hop, destination reachability, and PMTU are documented without assigning blame from one silent hop.
Stop whenProbe paths vary too widely for correlation or authorization excludes a method.
If this step fails
TCP trace reaches destination while default tracepath stops early.
Likely causeIntermediate policy filters UDP or TTL-expired replies but permits the service port.
traceroute -n -T -p 443 203.0.113.80curl --head --max-time 8 https://api.example.net/health
ResolutionTreat this as control-protocol filtering unless production evidence also fails; use the TCP/app path for diagnosis.
Security notes
- Do not enumerate ports; use only the incident service port.
Alternatives
- Use provider path telemetry or a remote capture when hops do not reply.
Stop conditions
- Do not escalate a silent hop without end-to-end corroboration.
verification
Test Path MTU without changing interface MTU
Use tracepath's PMTU result and controlled IPv4 Don't Fragment or IPv6 minimum-MTU probes to find the largest proven packet. Observe ICMP fragmentation-needed or Packet Too Big rather than lowering MTU blindly.
Why this step matters
PMTU black holes cause payload-dependent stalls and are often misdiagnosed as TLS or application timeouts. Probing without changing MTU preserves evidence and avoids broad performance impact.
What to understand
For Ethernet IPv4, 1472 bytes plus 20-byte IPv4 and 8-byte ICMP headers creates 1500. `-M do` forbids fragmentation and requests error feedback.
For IPv6, 1232 plus 40-byte IPv6 and 8-byte ICMPv6 headers produces the minimum 1280-byte packet.
Path MTU is direction- and route-specific; repeat from remote when return behavior matters.
System changes
- No persistent change; sends six controlled MTU-sized echo probes plus tracepath.
Syntax explained
-M do- Sets Don't Fragment behavior for IPv4 and fails rather than fragmenting locally.
-s 1472- Creates a 1500-byte IPv4 ICMP packet with standard headers.
-s 1232- Creates a 1280-byte IPv6 ICMPv6 packet.
Values stay on this page and are never sent or saved.
tracepath -n {{destinationIp}}; ping -4 -n -c 3 -M do -s 1472 {{destinationIp}}; ping -6 -n -c 3 -s 1232 {{destinationIPv6}}1?: [LOCALHOST] pmtu 1500 3: 203.0.113.80 18.202ms reached 3 packets transmitted, 3 received, 0% packet loss 3 packets transmitted, 3 received, 0% packet loss
Checkpoint: Prove or localize PMTU
Continue whentracepath PMTU and sized probes agree, or ICMP feedback identifies a smaller path without unexplained timeout.
Stop whenLarge packets vanish without error while small packets succeed.
If this step fails
IPv4 1472 fails with `message too long` locally.
Likely causeInterface/tunnel MTU is below 1500 before the first hop.
ip -details link show ens3ip route get 203.0.113.80tracepath -n 203.0.113.80
ResolutionDocument the actual egress/tunnel MTU and test appropriate payload; do not raise it beyond path capability.
Security notes
- Keep count low; do not sweep payload sizes against unapproved targets.
Alternatives
- Use application PLPMTUD tests for QUIC or protocols with their own discovery.
Stop conditions
- Known PMTU black hole requires network ownership before application tuning.
command
Capture only control packets for one failing transaction
After read-only evidence narrows the scope, create a root-only capture limited by interface, host, protocol, packet count, duration, and snapshot length. Capture only TCP SYN/FIN/RST and ICMP/ICMPv6 control traffic for the destination; reproduce one safe request and stop automatically.
Why this step matters
A narrow control-packet capture can distinguish no egress, no return, rejection, and ICMP error after read-only tests establish the exact interface and destination. Strict bounds minimize privacy and operational risk.
What to understand
Filter to one host, service port control flags, and ICMP/ICMPv6; payload-bearing established TCP packets are excluded.
Snapshot length 128 retains common headers/options and limited quoted error context, not full application streams.
Timeout and packet count are independent stop controls. Root-only directory and file protect metadata.
Check dropped-by-kernel count; a lossy capture cannot prove absence of packets.
System changes
- Creates `/var/tmp/oneliners-netdiag/control.pcap` containing up to {{capturePackets}} scoped packet headers for at most {{captureSeconds}} seconds.
Syntax explained
-i {{interfaceName}}- Captures only the route-selected interface.
-nn- Disables name and service lookups that would add traffic.
-s 128- Limits retained bytes per packet.
-c {{capturePackets}}- Stops after the maximum packet count.
-w file- Writes raw pcap with original timing for offline numeric reading.
BPF expression- Admits only destination TCP control flags and ICMP control traffic.
Values stay on this page and are never sent or saved.
sudo install -d -m 0700 /var/tmp/oneliners-netdiag && sudo timeout {{captureSeconds}} tcpdump -ni {{interfaceName}} -nn -s 128 -c {{capturePackets}} -w /var/tmp/oneliners-netdiag/control.pcap 'host {{destinationIp}} and ((tcp port {{destinationPort}} and (tcp[tcpflags] & (tcp-syn|tcp-fin|tcp-rst) != 0)) or icmp or icmp6)'tcpdump: listening on ens3, link-type EN10MB (Ethernet), snapshot length 128 bytes 14 packets captured 18 packets received by filter 0 packets dropped by kernel -rw------- 1 root root 2196 Jul 28 11:32 /var/tmp/oneliners-netdiag/control.pcap
Checkpoint: Accept minimal capture
sudo tcpdump -nn -r /var/tmp/oneliners-netdiag/control.pcap -c 20; sudo stat -c '%a %U %G %s %n' /var/tmp/oneliners-netdiag/control.pcapContinue whenOnly approved control packets appear, file is root-only, count/duration are bounded, and kernel captured without drops.
Stop whenUnexpected hosts, payload, interfaces, packet volume, or kernel drops appear.
If this step fails
Capture shows zero packets during a reproduced request.
Likely causeWrong interface/address family, application used a proxy/other address, off-host request, or BPF expression was too narrow.
ip route get 203.0.113.80 from 198.51.100.20getent ahosts api.example.netss -tnp dst 203.0.113.80
ResolutionStop and correct scope from route/socket evidence. Do not expand to unrestricted `any` capture.
Security notes
- Checksum and redact textual excerpts; never attach raw pcap to public issues.
Alternatives
- Use firewall counters or eBPF tracepoint metadata when pcap approval is unavailable.
Stop conditions
- Any privacy-scope breach stops capture immediately.
decision
Correlate packet direction with socket and application timing
Read the capture numerically, compare SYN/SYN-ACK/RST or ICMP timing with `ss -tin` and a curl timing breakdown, and classify the failure layer. Escalate with the smallest evidence set instead of changing route, DNS, MTU, firewall, and service simultaneously.
Why this step matters
Packet direction, socket state, and application timing are independent views of the same transaction. Together they classify the boundary without pretending a local host can identify every remote drop.
What to understand
Repeated SYN with no return and SYN-SENT proves local egress plus absent received response, not which upstream device dropped it.
RST localizes an active rejection, while SYN-ACK followed by TLS delay moves investigation above basic transport.
curl timing splits DNS, connect, TLS, first byte, and total. Compare with a healthy target and remote logs at matching timestamps.
Assign one owner and remediation hypothesis; changing multiple layers destroys causal evidence.
System changes
- No persistent network change; reads protected pcap, socket state, and sends one bounded HEAD health request.
Syntax explained
tcpdump -r -tttt- Reads numeric packet headers with full human-correlatable timestamps.
ss -tin dst- Shows TCP state and transport metrics for the exact peer.
curl --write-out- Prints stage-specific timing and selected remote address without response body.
Values stay on this page and are never sent or saved.
sudo tcpdump -nn -tttt -r /var/tmp/oneliners-netdiag/control.pcap; ss -tin dst {{destinationIp}}; curl --head --silent --show-error --output /dev/null --connect-timeout 3 --max-time 8 --write-out 'remote_ip=%{remote_ip} dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} first_byte=%{time_starttransfer} total=%{time_total} code=%{http_code}\n' https://{{destinationHost}}/health2026-07-28 11:32:04.122001 IP 198.51.100.20.48112 > 203.0.113.80.443: Flags [S], seq 121102003, win 64240, options [mss 1460,sackOK,TS val 91001 ecr 0,nop,wscale 7], length 0 2026-07-28 11:32:05.124110 IP 198.51.100.20.48112 > 203.0.113.80.443: Flags [S], seq 121102003, win 64240, options [mss 1460,sackOK,TS val 92003 ecr 0,nop,wscale 7], length 0 State SYN-SENT Recv-Q 0 Send-Q 1 Local Address:Port 198.51.100.20:48112 Peer Address:Port 203.0.113.80:443 remote_ip=203.0.113.80 dns=0.018 connect=0.000 tls=0.000 first_byte=0.000 total=3.001 code=000
Checkpoint: State an evidence-bounded diagnosis
Continue whenAt least two observations plus comparison support one failing layer and a named next owner; uncertainty is explicit.
Stop whenTimestamps, target, address, or evidence disagree, or conclusion exceeds what the capture can prove.
If this step fails
Capture shows SYN-ACK, but ss remains SYN-SENT.
Likely causeChecksum/offload display artifact, packet reached another namespace, firewall dropped inbound after capture hook, or tuple differs.
sudo nft list rulesetip netns listsudo tcpdump -ni ens3 -nn 'host 203.0.113.80 and tcp port 443' -c 10ss -nto state syn-sent
ResolutionCorrelate tuple and capture point with firewall/namespace trace; do not disable offloads or firewall without a separate test plan.
Security notes
- Publish a redacted timeline rather than full pcap.
Alternatives
- Correlate with remote capture, load-balancer request ID, or distributed trace.
Stop conditions
- Do not assign fault beyond the last evidenced boundary.
Finish line
Verification checklist
test -r /var/tmp/oneliners-netdiag/control.pcap && sudo tcpdump -nn -r /var/tmp/oneliners-netdiag/control.pcap -c 3The protected capture is readable, contains only approved destination control packets, and the change record includes timestamps, route, DNS, counters, probes, and application timing.ip route get {{destinationIp}} from {{localSourceIp}}; dig @{{resolverIp}} {{destinationHost}} A +noall +answer +stats; tracepath -n {{destinationIp}}The classified layer is supported by at least two independent observations and differs meaningfully from the healthy comparison.curl --head --fail --silent --show-error --connect-timeout 3 --max-time 8 https://{{destinationHost}}/health && ping -n -c 5 {{destinationIp}}The original safe transaction succeeds in its objective, packet loss/latency returns to baseline, and no unrelated state was changed on the diagnosing host.Recovery guidance
Common problems and safe checks
Gateway probes lose packets and interface RX/TX errors increase at the same time.
Likely causeLocal cable/optic/NIC/switch-port fault, speed/FEC mismatch, driver reset, or exhausted receive buffers.
sudo ethtool ens3sudo ethtool -S ens3 | grep -iE 'crc|error|miss|drop|reset'journalctl -k --since '-30 minutes' --no-pager
ResolutionEscalate with exact counter deltas, driver/firmware, switch port, and timestamps. Move traffic only through an approved redundant path; do not reset counters before capture.
DNS returns quickly, but it resolves the hostname to an unexpected address.
Likely causeSplit horizon, stale recursive cache, short-lived load-balancer rotation, search domain, or wrong resolver link.
resolvectl status --no-pagerdig @192.0.2.53 api.example.net A +noall +answer +commentsdig api.example.net CNAME +trace
ResolutionCompare configured and authoritative answers with TTL/time and application cache. Correct DNS ownership rather than pinning `/etc/hosts` as a hidden workaround.
Ping shows loss at hop 2, but destination ping and application are healthy.
Likely causeThe intermediate router rate-limits or deprioritizes TTL-expired/echo replies while forwarding transit packets normally.
traceroute -n -I -q 5 203.0.113.80traceroute -n -T -p 443 -q 5 203.0.113.80ping -n -c 20 203.0.113.80
ResolutionDo not attribute loss to that hop. Use end-to-end loss, following-hop behavior, transport tests, and network telemetry.
SYN packets leave repeatedly, but no SYN-ACK, RST, or ICMP returns.
Likely causeUpstream ACL/firewall, wrong route, asymmetric return, destination listener absence behind silent policy, or remote black hole.
ip route get 203.0.113.80 from 198.51.100.20sudo tcpdump -ni ens3 -nn -c 20 'host 203.0.113.80 and tcp port 443'traceroute -n -T -p 443 203.0.113.80
ResolutionCorrelate a remote/firewall log or capture at the next boundary. The local capture proves egress and absence of return, not the exact remote drop point.
TCP connects and TLS completes, but first byte is slow.
Likely causeApplication queue, upstream backend, database, server CPU, TLS inspection, or response generation rather than network establishment.
curl --output /dev/null --silent --write-out '%{time_connect} %{time_appconnect} %{time_starttransfer} %{time_total}\n' https://api.example.net/healthss -tin dst 203.0.113.80
ResolutionHand the timing split and trace correlation to the application/service owner; stop expanding network capture when connect/TLS are healthy.
Reference
Frequently asked questions
Does packet loss in traceroute identify the bad router?
Not necessarily. Routers often rate-limit replies to probes while forwarding transit traffic. End-to-end loss and subsequent-hop behavior are more meaningful.
Should I flush DNS first?
No. Capture resolver, answer, TTL, application behavior, and comparison before changing cache state. Flushing can remove the evidence and affect unrelated applications.
Why not capture on `any` with snap length 0?
That collects far more interfaces, flows, and payload than needed. Route selection tells you the interface; a narrow filter and bounded snap length minimize privacy and analysis cost.
Can lowering MTU fix a hanging connection?
It may mask a PMTU or tunnel problem, but changes every flow and can hide blocked ICMP. First observe tracepath and Packet Too Big/fragmentation-needed behavior.
Why test a literal IP and a hostname?
The difference separates DNS/address selection from transport/application reachability, while preserving the original Host/SNI contract for the hostname test.
Recovery
Rollback
The diagnostic path is read-only until the scoped capture, which creates one protected pcap and directory. Rollback therefore means stop probes, remove or transfer evidence according to policy, and revert no network configuration. Any remediation belongs to a separately reviewed change with its own rollback.
- Stop any still-running timeout/tcpdump process and verify no broad capture remains with `pgrep -a tcpdump`.
- If evidence is not required, securely remove `/var/tmp/oneliners-netdiag/control.pcap` and its directory; if it is required, checksum, encrypt, restrict, and transfer it through the approved incident channel.
- Clear no DNS cache, route, neighbor, firewall, MTU, sysctl, or service state because this guide did not change them; document any manual deviation separately.
- After another team performs remediation, rerun the original bounded test contract and comparison rather than expanding capture scope immediately.
Evidence