Chapter 54 — VPNs and tunnels: WireGuard, IPsec, GRE, overlays, and MTU traps

In one sentence

A tunnel puts one packet inside another packet. The price is extra headers, new routing rules, a smaller effective MTU, and one more place where things can break silently.

The problem

You need two private networks to talk across the internet, or containers on different hosts to share one flat network, or remote laptops to reach internal services. You can't change the internet's routing, so you encapsulate: wrap your packet in another packet whose addresses the internet can route, and unwrap it at the far end.

The tunnel family

Technology	Encapsulation	Encrypted?	What it is good for
WireGuard	IP inside UDP (port 51820 by default)	Yes (Noise protocol, ChaCha20-Poly1305)	Simple site-to-site or host VPN, modern crypto, small config
IPsec (ESP)	IP inside ESP (IP protocol 50), or UDP 4500 behind NAT	Yes	Enterprise/firewall VPNs, cloud site-to-site VPNs; standard but complex
GRE / IP-in-IP	IP inside IP (protocol 47 / 4)	No	Simple encapsulation, often combined with IPsec
VXLAN / Geneve	Ethernet frame inside UDP (4789 / 6081)	No	Data-centre and container overlays; many virtual L2 networks over one L3 underlay
SSH tunnel	TCP inside TCP	Yes	Quick ad-hoc port forwarding, not a network design

How it actually works

WireGuard in four ideas:

Each peer has a key pair. Peers are identified only by public key; there are no usernames or certificates.
AllowedIPs is both routing table and access list. Outbound: "send packets for 10.20.0.0/24 to this peer". Inbound: "accept packets from this peer only if their source is in 10.20.0.0/24". The WireGuard paper calls this cryptokey routing.
It's UDP and connectionless. A handshake happens on demand and is refreshed every two minutes. When a peer's IP address changes (a laptop moving from Wi-Fi to 4G), the tunnel follows it automatically.
Behind NAT, set PersistentKeepalive = 25 so the NAT mapping (Chapter 12) doesn't expire while the tunnel is idle.

IPsec has two phases: IKE (UDP 500, or 4500 behind NAT) authenticates the peers and negotiates keys, creating Security Associations. ESP then encrypts each packet. Tunnel mode wraps the whole original packet (VPNs). Transport mode protects only the payload (host to host). Most "IPsec is down" incidents are mismatched IKE proposals, expired pre-shared keys, or UDP 500/4500 blocked somewhere.

VXLAN carries a whole Ethernet frame, so the two ends look like they're on the same switch even if they're in different racks or regions. A 24-bit VNI separates up to 16 million virtual networks. This is how many Kubernetes CNIs and cloud VPCs build "one flat network" on top of routed infrastructure (OSI guide, Layer 2).

The MTU tax

Every tunnel adds headers, and those bytes come out of the inner packet's space:

Tunnel (over IPv4, 1500-byte path)	Overhead	Inner MTU
IP-in-IP	20	1480
GRE	24	1476
VXLAN	50	1450
WireGuard	60 (IPv4) / 80 (IPv6)	1420 (default, safe for both)
IPsec ESP tunnel mode	~50–73 (varies with cipher, padding)	~1400–1438
PPPoE (not a tunnel you chose, but the same effect)	8	1492

If the inner packet is still 1500 bytes, something must shrink, fragment, or fail. Common fingerprints:

SSH works, file transfer stalls.
Small HTTP responses work, large ones hang.
VPN connects, but some sites or APIs fail.
ping works until you add "don't fragment" and a larger size.

The two fixes:

Set the inner MTU correctly on the tunnel interface (WireGuard defaults to 1420 for this reason).
Clamp TCP MSS on the router so TCP never tries to send oversized segments. This is one rule, and it's the standard fix for PPPoE and site-to-site VPNs:

sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

MSS clamping only helps TCP. UDP protocols (QUIC, DNS, games) still depend on a correct MTU or working PMTU discovery.

Practice (30 min): two tunnels between two namespaces

Run on a Linux lab machine or a privileged container (docker run --rm -it --privileged debian:bookworm, then apt-get install -y iproute2 iputils-ping wireguard-tools). Two namespaces, a and b, connected by a veth "internet":

ip netns add a; ip netns add b
ip link add va type veth peer name vb; ip link set va netns a; ip link set vb netns b
ip -n a addr add 192.0.2.1/24 dev va; ip -n a link set va up; ip -n a link set lo up
ip -n b addr add 192.0.2.2/24 dev vb; ip -n b link set vb up; ip -n b link set lo up

# Tunnel 1: IP-in-IP (no encryption)
ip -n a link add tun0 type ipip local 192.0.2.1 remote 192.0.2.2
ip -n b link add tun0 type ipip local 192.0.2.2 remote 192.0.2.1
ip -n a addr add 10.10.0.1/30 dev tun0; ip -n a link set tun0 up
ip -n b addr add 10.10.0.2/30 dev tun0; ip -n b link set tun0 up
ip -n a link show tun0                                # mtu 1480
ip netns exec a ping -c1 -M do -s 1452 10.10.0.2      # 1452 + 8 ICMP + 20 IP = 1480: fits
ip netns exec a ping -c1 -M do -s 1453 10.10.0.2      # one byte more: "message too long"

# Tunnel 2: WireGuard (encrypted)
cd /tmp; wg genkey | tee a.key | wg pubkey > a.pub; wg genkey | tee b.key | wg pubkey > b.pub
ip -n a link add wg0 type wireguard
ip netns exec a wg set wg0 private-key a.key listen-port 51820 \
  peer $(cat b.pub) allowed-ips 10.20.0.2/32 endpoint 192.0.2.2:51820
ip -n b link add wg0 type wireguard
ip netns exec b wg set wg0 private-key b.key listen-port 51820 \
  peer $(cat a.pub) allowed-ips 10.20.0.1/32 endpoint 192.0.2.1:51820
ip -n a addr add 10.20.0.1/24 dev wg0; ip -n a link set wg0 up
ip -n b addr add 10.20.0.2/24 dev wg0; ip -n b link set wg0 up

ip netns exec a ping -c2 10.20.0.2
ip netns exec a wg show wg0                           # latest handshake, transfer counters
ip netns exec a ping -c1 -M do -s 1392 10.20.0.2      # 1392 + 28 = 1420: the WireGuard MTU

All of this was run while writing this guide. The MTUs came out at exactly 1480 and 1420, and wg show reported latest handshake: 1 second ago.

Now capture on the "internet" link while you ping through each tunnel: ip netns exec a tcpdump -ni va. The IP-in-IP packets show the inner ICMP in plain text. The WireGuard packets are opaque UDP datagrams to port 51820. That's the difference encryption makes.

Design advice

Document the underlay path, overlay prefixes, and MTU for every tunnel.
Prefer route-based VPNs (a tunnel interface plus routes) over policy-based ones when you need several networks and failover.
Monitor tunnel state and loss and latency through it, not just "is the process running?". For WireGuard, alert when the latest handshake is older than 3 minutes on a tunnel that should be busy.
Keep tunnel routes specific. Default-route VPNs are easy to misuse and make every outage look like a VPN outage.
Test with real payload sizes after every change: a large HTTP download through the tunnel, not just ping.

Common confusions

"VXLAN is a VPN." It's an overlay with no encryption. Traffic between hosts is readable unless you add IPsec/WireGuard underneath, or mTLS above it.
"WireGuard is connection-oriented like TCP." It's UDP with no connection state you can see. "Connected" just means a recent handshake.
"Fragmentation will sort it out." Most tunnels set DF on the outer packet or can't fragment the inner one, and ICMP "too big" messages are often filtered. Assume fragmentation will not save you.

Check yourself

What does AllowedIPs do in each direction?
Why does WireGuard default to an MTU of 1420 instead of 1500?
What does MSS clamping fix, and what can't it fix?
You can SSH through a new VPN but git clone hangs. What's your first hypothesis and your first test?