Appendix A — Troubleshooting playbooks (symptom → cause → fix)

General method: work up the stack. Link (carrier/errors) → IP (address/route/ARP) → transport (handshake/RST/retransmit) → app (status codes/ latency). Capture on both ends for anything about loss or latency. Change one thing at a time. Note the exact error string — it encodes the layer (§24.7).

A1. "Connection timed out" (no response at all)

A2. "Connection refused" (ECONNRESET/ECONNREFUSED immediately)

• RST to your SYN → nothing listening on that port. Verify: ss -tlnp on the server; is it bound to 127.0.0.1 only (not 0.0.0.0/::)? Right port? Right container/netns? Service crashed?
• Connect succeeds then instant RST → accept-queue overflow (nstat | grep ListenOverflows, raise listen() backlog + somaxconn), or app crashes on accept(), or an IPS resetting.

A3. "Works then hangs on big transfers / TLS stalls"

Signature of MTU/PMTUD black hole (Ch. 8). tcpdump shows a ~1500 B packet retransmitted with no ACK. Fixes: unblock ICMP type 3 / ICMPv6 type 2; MSS clamp on the tunnel (iptables ... TCPMSS --clamp-mss-to-pmtu); sysctl net.ipv4.tcp_mtu_probing=1; lower interface/pod MTU. In K8s: check pod MTU vs underlay (Ch. 55).

A4. Intermittent latency spikes at exactly 200 ms / 1 s / 3 s

• 200 ms / 1 s flat floor on request/response → Nagle × delayed-ACK (Ch. 33). Fix: TCP_NODELAY on the client, or write the whole request in one send().
• 200 ms tail on the last segment of small responses → RTO after tail loss (Ch. 28). Fix: RACK-TLP (newer kernel), ip route ... rto_min 20ms in the DC.
• 1 s / 3 s connect latency → SYN or final-ACK loss / accept-queue overflow (nstat | grep -E 'ListenDrops|ListenOverflows|SynRetrans').
• 5 s DNS stalls → a dead server in resolv.conf; add options timeout:1 attempts:2 rotate or run a local caching resolver (Ch. 20.8).

A5. Throughput far below link rate, no packet loss

1. ss -tie dst <peer> → is it app_limited? → fix the app (single-threaded producer, small writes, computing Content-Length by buffering).
2. Window scaling on both SYNs? (tcpdump the handshake). Middlebox stripping options? (Ch. 23 war story).
3. buffer_max ≥ 2 × BDP? Compute BDP = bandwidth × RTT (Ch. 35). Raise net.ipv4.tcp_rmem/tcp_wmem max and net.core.rmem_max/wmem_max.
4. iperf3 -P8 ≫ single stream? → CC/window-limited; consider BBR (Ch. 32) or accept parallelism.
5. Per-flow cloud cap (TGW/NAT GW/VPN/single-flow ~5 Gbit/s)? → parallel streams (Ch. 57).
6. RX drops? ethtool -S, /proc/net/softnet_stat, IRQ affinity (Ch. 48).

A6. Throughput low with packet loss

• Confirm loss is real & directional: two-ended capture, compare by seq; mtr loss must persist to the final hop.
• Link errors: ethtool -S rx_crc_errors/rx_errors; check cabling/SFP/optics, duplex mismatch (Ch. 5).
• Reordering masquerading as loss? nstat | grep -Ei 'reorder|dsack|spurious'; fix LACP/ECMP hash to per-flow (Ch. 27).
• If loss is inherent (Wi-Fi, satellite, transoceanic): SACK on (verify), switch senders to BBR, or use parallel streams / FEC. Mathis: throughput ≈ MSS/(RTT·√p) (Ch. 36).
• Bufferbloat check: does RTT balloon under load? → AQM (fq_codel/cake) at the bottleneck (Ch. 44).

A7. Connections die after N seconds/minutes of idle

Middlebox idle timeout reaping the flow (Ch. 34, 53). Common values: 350 s (AWS NLB/NAT GW), 60 s (some ELB/proxies), 300 s (consumer NAT), 5–15 min (enterprise FW). Fix: application heartbeat / TCP keepalive with idle interval shorter than the smallest timeout on the path (e.g. 30–60 s), and TCP_USER_TIMEOUT for bounded failure.

A8. File-descriptor leak / too many open files

• ss -tan state close-wait | wc -l climbing → your app isn't calling close() on a code path (Ch. 25.5). Find it (error/exception branches).
• ss -tan state fin-wait-2 climbing → the peer isn't closing; bounded by tcp_fin_timeout.
• Legit high count → raise ulimit -n / LimitNOFILE= / fs.file-max.

A9. Lots of TIME_WAIT on a client

Normal if you open many short connections. Only act if near port exhaustion (connect() → EADDRNOTAVAIL): enable net.ipv4.tcp_tw_reuse=1, widen ip_local_port_range, and — the real fix — use connection pooling / keep-alive (Ch. 21 war story). Never tcp_tw_recycle.

A10. UDP app "losing data"

nstat | grep -Ei 'UdpInErrors|UdpRcvbufErrors|UdpNoPorts'. RcvbufErrors → receiver too slow / buffer too small: raise SO_RCVBUF + net.core.rmem_max, use recvmmsg(), add a drain thread. InErrors/NoPorts → nothing listening / checksum failures. Remember UDP loss is silent — you must instrument it (Ch. 22).

A11. DNS is "slow" or "flaky"

• dig +trace <name> — where does it break? NXDOMAIN cached? (SOA minimum TTL).
• Capture UDP/53: how many queries per lookup? ndots:5 + search → 4–5 queries; use FQDN with trailing dot (Ch. 20.8).
• Large response truncated (TC flag) → is TCP/53 allowed? IP fragments dropped? EDNS buffer too big? Set 1232 (Ch. 15).
• Per-resolver throttle (cloud .2 resolver ~1024 pps/ENI) → NodeLocal DNSCache / local unbound.
• DoH in the browser bypassing internal resolver → internal names fail (Ch. 20.7).

A12. TLS handshake fails / times out for some clients

• openssl s_client -connect host:443 -servername host — which step fails?
• Cert chain incomplete (missing intermediate)? openssl ... -showcerts and test from a machine with a fresh trust store.
• ClientHello spans two segments (PQC key shares, many extensions) and a middlebox/LB only reads the first packet → handshake hangs (Ch. 38 war story).
• Protocol/cipher mismatch (old client, TLS 1.0 disabled), SNI required but not sent, clock skew (cert "not yet valid").

A13. Kubernetes: pod can't reach Service / another pod

• Pod→pod same node works, cross-node fails → CNI overlay / route issue: tcpdump for VXLAN/GENEVE (udp 4789/6081) on the node eth0; check tunnel offloads (Ch. 56 war story); check node-to-node SG allows the overlay port.
• Pod→Service fails but pod→podIP works → kube-proxy/eBPF DNAT rules or Endpoints empty (readiness failing); iptables-save | grep <svc-clusterip>.
• Big responses hang → pod MTU vs underlay (Ch. 55 war story).
• Cross-node blocked → NetworkPolicy default-deny; cilium monitor / calicoctl to see drops.
• DNS slow → ndots:5, CoreDNS pods, conntrack pressure (Ch. 56).

A14. "The network is dropping packets" (general)

Before opening a carrier ticket, rule out:

• NIC/driver: ethtool -S, /proc/net/softnet_stat, ring sizes, IRQ affinity.
• Socket buffers / accept queue / conntrack table: nstat -az | grep -Ei 'drop|overflow|prune|nf_conntrack', conntrack -S, dmesg.
• qdisc drops (may be intentional AQM): tc -s qdisc.
• Firewall INVALID/policy drops: rule counters.
• Application 4xx/5xx mislabeled as "network" (Ch. 42 war story) — check at the packet layer whether it's RST/timeout (transport) or a clean HTTP status (app).

Appendix B — Port and protocol quick reference

B1. IP protocol numbers (IPv4 Protocol / IPv6 Next Header)

B2. Common TCP/UDP ports

B3. ICMP / ICMPv6 types worth knowing

B4. EtherTypes

Appendix C — Subnetting cheat sheet

C1. Prefix ↔ mask ↔ hosts

Mask octet values: /25→.128 /26→.192 /27→.224 /28→.240 /29→.248 /30→.252.

C2. The 4-step method (recap from Ch. 11)

Given A.B.C.D /p:

1. Interesting octet = ceil(p/8); block size = 256 − mask_octet in that octet.
2. Network = round D (or the interesting octet) down to a multiple of the block size.
3. Broadcast = network + block size − 1 (in that octet), host bits all 1.
4. Usable = network+1 … broadcast−1 (except /31, /32).

C3. Worked examples

C4. VLSM allocation rule

Sort requirements largest first, allocate from the top of the block, next subnet starts at previous network + its block size. (Allocating small subnets first fragments the space — Ch. 11.3.)

C5. IPv6 quick facts

• LAN = /64 always (SLAAC assumes it).
• Site from ISP: /48 (65 536 /64s) or /56 (256 /64s).
• P2P link: still use a /64 (or /127 per RFC 6164 to dodge a ping-pong attack).
• ::1 loopback, fe80::/10 link-local (always present), fd00::/8 ULA (private), 2000::/3 global, ff00::/8 multicast, 2001:db8::/32 docs.
• No broadcast; no ARP (NDP via ICMPv6); routers never fragment.

Appendix D — tcpdump / ss / ip command cookbook

D1. ip (iproute2)

ip -br -c addr                     # brief, colored: interfaces + IPs
ip -4 addr show dev eth0
ip -6 addr show
ip link set eth0 up / down
ip link set eth0 mtu 9000
ip link set eth0 txqueuelen 1000
ip route                            # main routing table
ip route get 8.8.8.8               # exact forwarding decision (src, dev, gw, mtu)
ip route add 10.0.0.0/8 via 10.1.1.1 dev eth0
ip route add default via 192.168.1.1
ip route add 10.9.0.0/16 via 10.1.1.1 mtu 1400        # pin MTU
ip route flush cache
ip rule                             # policy routing rules
ip rule add from 192.168.50.0/24 table 100
ip route add default via 10.0.0.1 table 100
ip neigh                            # ARP / NDP cache + states
ip neigh flush all
ip -s link show eth0               # RX/TX packets, errors, dropped
ip -s -s link show eth0           # + detailed error breakdown
ip netns list ; ip netns exec red <cmd>
ip -d link show vxlan0            # decode VXLAN/GRE/bond details
ip maddr                           # multicast group memberships

D2. ss

ss -tan                            # all TCP, numeric
ss -tanp                           # + process (root for others)
ss -tlnp                           # listening TCP + owner
ss -uanp                           # UDP
ss -s                              # summary by state
ss -tan state established
ss -tan state time-wait | wc -l
ss -tan '( dport = :443 or sport = :443 )'
ss -tan dst 10.0.0.0/8
ss -tie                            # extended + internal TCP info (rtt, cwnd, retrans, CC)
ss -tim                            # + memory per socket
ss -tio                            # + timers (retransmit/keepalive countdown)
ss -tan state syn-recv | wc -l    # SYN flood indicator
ss -tanp | awk '{print $1}' | sort | uniq -c    # state histogram
ss -K dst 10.1.2.3                # kill matching sockets (root; needs CONFIG_INET_DIAG_DESTROY)

D3. tcpdump

# Capture to file, headers only, kernel buffer bumped:
sudo tcpdump -ni eth0 -s128 -B 8192 -w /tmp/c.pcap 'tcp port 443 and host 1.2.3.4'
# Rotating ring (24 x 100MB) for intermittent bugs:
sudo tcpdump -ni eth0 -s0 -C 100 -W 24 -w /tmp/ring.pcap 'port 443'
# Read back with filters:
tcpdump -nr /tmp/c.pcap 'tcp[tcpflags] & tcp-rst != 0'
# Handshakes & resets only:
sudo tcpdump -ni any '(tcp[tcpflags] & (tcp-syn|tcp-rst)) != 0'
# Retransmit hunting is easier in tshark:
tshark -nr /tmp/c.pcap -Y tcp.analysis.retransmission
tshark -nr /tmp/c.pcap -q -z conv,tcp
tshark -nr /tmp/c.pcap -q -z io,phs             # protocol hierarchy
tshark -nr /tmp/c.pcap -q -z expert
# HTTP requests with timing:
tshark -nr /tmp/c.pcap -Y http.request -T fields -e ip.dst -e http.host -e http.request.uri
# DNS:
sudo tcpdump -ni any -s0 'udp port 53' -w /tmp/dns.pcap
# ARP / NDP:
sudo tcpdump -ni eth0 arp
sudo tcpdump -ni eth0 'icmp6 && ip6[40] >= 133 && ip6[40] <= 137'
# VXLAN / GENEVE (decodes inner packet in Wireshark):
sudo tcpdump -ni eth0 'udp port 4789 or udp port 6081'
# Fragments only:
sudo tcpdump -ni any 'ip[6:2] & 0x3fff != 0'
# By TCP payload prefix ("GET "):
sudo tcpdump -ni any 'tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420'

D4. Diagnostics

mtr -rwzbc100 <host>                   # loss/latency per hop + ASN, report mode
traceroute -n <host> ; traceroute -T -p 443 -n <host> ; tracepath <host>
ping -c5 <host> ; ping -M do -s 1472 <host>       # reachability ; PMTU probe
nstat -az                              # every kernel net counter
nstat ; sleep 10 ; nstat               # deltas over 10s
cat /proc/net/snmp /proc/net/netstat /proc/net/sockstat
ethtool eth0 ; ethtool -S eth0 ; ethtool -k eth0 ; ethtool -g eth0 ; ethtool -l eth0
tc -s qdisc show dev eth0 ; tc -s class show dev eth0
conntrack -L ; conntrack -S ; sysctl net.netfilter.nf_conntrack_count
iperf3 -s   /   iperf3 -c <host> -t30 -P8 -C bbr
curl -w 'dns %{time_namelookup} conn %{time_connect} tls %{time_appconnect} ttfb %{time_starttransfer} total %{time_total}\n' -o /dev/null -s https://<host>
dig +trace <name> ; dig @1.1.1.1 <name> ; dig -x <ip> ; dig <name> HTTPS
openssl s_client -connect <host>:443 -servername <host> -brief </dev/null
ss -tie dst <host>                      # the single most useful per-connection view

D5. Sysctl inspection

sysctl -a --pattern 'net\.(ipv4|core|ipv6)\.' | less
sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc
sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem net.core.rmem_max net.core.wmem_max
sysctl net.ipv4.ip_local_port_range net.core.somaxconn net.ipv4.tcp_max_syn_backlog
sysctl net.ipv4.tcp_syncookies net.ipv4.tcp_tw_reuse net.ipv4.tcp_slow_start_after_idle
sysctl net.netfilter.nf_conntrack_max net.netfilter.nf_conntrack_count

Appendix E — Glossary

AIMD — Additive Increase, Multiplicative Decrease. TCP's congestion-window control law; source of fairness and the "sawtooth."

ALG — Application Layer Gateway. NAT helper that rewrites addresses embedded in payloads (FTP, SIP). Often buggy.

Anycast — one IP prefix announced from many locations; BGP routes each client to the nearest.

ARP — Address Resolution Protocol. Maps an IPv4 address to a MAC on the local link. IPv6 uses NDP.

AQM — Active Queue Management (CoDel, PIE, fq_codel, CAKE). Keeps bottleneck queues short to fight bufferbloat.

AS / ASN — Autonomous System / its number. A network with one routing policy; the unit BGP connects.

BDP — Bandwidth-Delay Product = bandwidth × RTT. Bytes "in flight" needed to fill a path; the target window size.

BGP — Border Gateway Protocol. Inter-AS path-vector routing; runs the internet's route table over TCP/179.

BBR — Bottleneck Bandwidth and RTT. Model-based congestion control that paces to measured bandwidth and largely ignores loss.

Bufferbloat — excessive latency from oversized, unmanaged buffers at a bottleneck being kept full by loss-based TCP.

CGNAT — Carrier-Grade NAT (RFC 6598, 100.64.0.0/10). A second NAT layer at the ISP; breaks port forwarding and IP-based identity.

CIDR — Classless Inter-Domain Routing. Explicit prefix lengths (10.0.0.0/8); replaced address classes in 1993.

conntrack — the Linux connection-tracking table used by stateful firewalling and NAT.

CUBIC — Linux's default congestion control; window grows as a cubic function of time since the last loss.

cwnd / rwnd / ssthresh — congestion window / receive (advertised) window / slow-start threshold.

DHCP — Dynamic Host Configuration Protocol. DORA exchange assigns IP, mask, gateway, DNS, lease.

DNS — Domain Name System. Hierarchical, cached name→record database (A/AAAA/CNAME/MX/NS/SOA/TXT/SRV/HTTPS/…).

DSCP — Differentiated Services Code Point. 6-bit QoS class in the IP header (EF, AFxx, CS0…).

ECMP — Equal-Cost Multi-Path. Per-flow hashing across equal-cost next hops; caps a single flow to one path.

ECN — Explicit Congestion Notification. Routers mark (not drop) packets to signal congestion; endpoints react without loss.

EUI-64 — deriving a 64-bit IPv6 interface ID from a 48-bit MAC (now usually replaced by random privacy addresses).

FIN / RST / SYN / ACK — TCP flags: finish (graceful), reset (abort), synchronize (setup), acknowledge.

GRO / GSO / TSO / LRO — Generic Receive / Generic Segmentation / TCP Segmentation / Large Receive Offload. Batch packets to cut per-packet CPU.

Happy Eyeballs — algorithm that races IPv6 and IPv4 connection attempts so a broken family costs ~250 ms, not a timeout.

HOL blocking — Head-of-Line blocking. One stalled item delays everything queued behind it (TCP under HTTP/2; fixed by QUIC streams).

ICMP / ICMPv6 — control/error messaging for IP (echo, unreachable, time exceeded, packet-too-big). ICMPv6 also carries NDP.

ISN — Initial Sequence Number. Randomized per connection to resist off-path injection.

MSS / MTU — Maximum Segment Size (TCP payload per segment) / Maximum Transmission Unit (L3 payload per frame). MSS ≈ MTU − IP − TCP headers.

Mathis equation — throughput ≈ MSS / (RTT · √loss) for a loss-based TCP flow.

NAT / PAT — Network Address (and Port) Translation. Many private hosts share one public IP by rewriting the 4-tuple.

NDP — Neighbor Discovery Protocol (ICMPv6). Replaces ARP + router discovery + redirects for IPv6 (types 133–137).

PMTUD — Path MTU Discovery. Learn the smallest link on a path via "packet too big" ICMP; broken when ICMP is filtered (black hole).

QUIC — a reliable, multiplexed, always-encrypted transport over UDP; foundation of HTTP/3. Independent streams, connection migration, 0-RTT.

RACK-TLP — Recent ACKnowledgment loss detection + Tail Loss Probe. Time-based, reordering-robust replacement for dup-ACK counting.

RPKI / ROA / ROV — Resource PKI / Route Origin Authorization / Route Origin Validation. Cryptographic defense against BGP prefix hijacks.

RTO / SRTT / RTTVAR — Retransmission Timeout and the smoothed RTT / RTT variance estimators that set it.

SACK / DSACK — Selective ACK (report received ranges above a gap so the sender retransmits only holes) / Duplicate SACK (report data received twice).

SLAAC — StateLess Address AutoConfiguration. IPv6 hosts self-assign addresses from a Router Advertisement prefix.

SNI — Server Name Indication. The hostname in the TLS ClientHello so one IP serves many certificates (plaintext unless ECH).

STUN / TURN / ICE — discover your NAT mapping / relay when direct fails / the algorithm that tries all candidate paths. NAT traversal for P2P and WebRTC.

TCB — Transmission Control Block. Per-connection kernel state (tuple, seq numbers, windows, timers, buffers).

TIME_WAIT — post-close state (2×MSL) on the side that sent the last ACK; absorbs stragglers and prevents tuple confusion.

TTL / Hop Limit — per-hop decremented counter; at 0 the packet is dropped and ICMP "time exceeded" returned (powers traceroute).

uRPF / BCP 38 — Unicast Reverse Path Forwarding / the practice of ingress filtering spoofed source addresses.

VLAN / VXLAN — Virtual LAN (802.1Q, 12-bit ID, one switch) / Virtual Extensible LAN (24-bit VNI, MAC-in-UDP overlay across an IP underlay).

Window scaling — TCP option multiplying the 16-bit window by up to 2^14, so windows can exceed 64 KB (required for high-BDP paths).