8. MTU, fragmentation triggers, and jumbo frames

8.1 The MTU chain

MTU is the largest Layer-3 payload a link will carry. The effective end-to-end MTU is the minimum of every link on the path — the Path MTU (PMTU).

8.2 What happens when a packet is too big

• IPv4, DF (Don't Fragment) bit = 0: the router fragments the packet (Ch. 15). Slow, fragile, disables some hardware offloads.
• IPv4, DF = 1: router drops it and sends back ICMP type 3 code 4 "Fragmentation needed" including the next-hop MTU.
• IPv6: routers never fragment. The source must; an oversize packet gets ICMPv6 type 2 "Packet Too Big".

TCP always sets DF and relies on PMTUD (Path MTU Discovery): start at the local MTU, shrink when a "too big" ICMP arrives, cache per-destination.

8.3 The PMTUD black hole

If a firewall blocks all ICMP (a depressingly common "hardening" mistake), the "fragmentation needed" messages never arrive. Symptom: the TCP handshake succeeds (small packets), then the connection hangs the instant a full-size data packet is sent — it's silently dropped forever. Classic signature: curl connects, sends the request, and stalls; small pages work, big ones don't; ping works but ping -s 1472 -M do fails.

Fixes:

• Allow ICMP type 3 (and ICMPv6 type 2) through firewalls. Always.
• MSS clamping on the tunnel/router: iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu. This rewrites the MSS option in SYN packets so endpoints never send oversize segments in the first place.
• Linux ip route ... mtu lock 1400 for a known-bad path.
• PLPMTUD (RFC 4821 / RFC 8899) — probe for MTU using the transport itself, no ICMP needed. QUIC uses this.

8.4 Jumbo frames

Setting MTU 9000 on every device in a data-center segment means ~6× fewer packets per MB → less CPU, higher throughput, especially for storage (iSCSI, NFS) and backups. Rules:

• Every device on the L2 segment must agree, including the switches. One device at 1500 that receives a 9000 frame just drops it.
• Doesn't help across the internet (path will have a 1500 link).
• Test: ping -M do -s 8972 <peer> (8972 + 8 ICMP + 20 IP = 9000).

8.5 War story

A new site-to-site VPN worked for SSH and ping but every git clone and file copy froze at a few KB. tcpdump showed a 1500-byte packet leaving repeatedly with no ACK. The firewall vendor's "recommended" ruleset dropped all ICMP. Adding an allow rule for ICMP type 3, plus MSS clamping to 1400 on the tunnel interface, fixed every stuck transfer. Lesson: "connects then hangs on big transfers" ≈ MTU/PMTUD black hole, 95% of the time.

Try it yourself

# Find the path MTU to a host (largest non-fragmented ICMP payload)
ping -M do -s 1472 example.com          # 1472+8+20 = 1500; shrink until it passes
tracepath example.com                    # reports PMTU per hop, no root needed
ip route get 8.8.8.8                      # shows the MTU that will be used
ip route add 10.0.0.0/8 via 10.1.1.1 mtu 1400   # pin an MTU for a route