56. Overlays: VXLAN, and how Kubernetes pods talk

56.1 The pod-network requirement

Kubernetes demands: every pod gets a unique IP, and every pod can reach every other pod without NAT (across nodes). The underlay (the VPC/physical network) usually doesn't know about pod IPs, so the CNI plugin must bridge that gap. Two broad approaches:

56.2 Approach A — routed / native (no encapsulation)

The pod CIDR is made routable on the underlay:

Cloud-native CNIs (AWS VPC CNI, Azure CNI, GKE): pods get real VPC IPs from ENIs/secondary ranges. No overlay, MTU stays 1500 (or jumbo), the VPC router forwards pod↔pod directly. Cost: consumes VPC address space, ENI/IP limits per node.
BGP (Calico in BGP mode, Cilium BGP, kube-router): each node advertises its local pod /26 via BGP to the ToR/underlay. Pure L3, no encap, easy to troubleshoot with normal tools. Needs BGP on the fabric.

56.3 Approach B — overlay (encapsulation)

Wrap each pod packet inside a new IP/UDP packet addressed node→node, so the underlay only ever sees node IPs.

VXLAN (the common one; Flannel default, Calico VXLAN mode, Cilium VXLAN):

[ outer Eth ][ outer IP: nodeA -> nodeB ][ outer UDP dport 4789 ]
   [ VXLAN header: 8 bytes, 24-bit VNI ]
      [ inner Eth ][ inner IP: podA -> podB ][ inner payload ]

VNI (VXLAN Network Identifier, 24 bits → 16M segments) separates virtual networks — the multi-tenant successor to the 12-bit VLAN ID.
A VTEP (VXLAN Tunnel Endpoint) on each node encapsulates/decapsulates. It learns "pod MAC/IP → remote node IP" mappings via a control plane (the CNI watching the K8s API), or historically multicast/flood-and-learn.
Overhead = 50 bytes (14 inner Eth + 8 VXLAN + 8 UDP + 20 outer IP) → pod MTU 1450 on a 1500 underlay. GENEVE (Cilium default, OVN) is similar with TLV options → slightly more overhead. IP-in-IP (Calico) = 20 bytes → MTU 1480. WireGuard encryption (Calico/Cilium encryption) = ~60–80 bytes.

56.4 Performance considerations

Offloads must be tunnel-aware. Without VXLAN-aware checksum/TSO/GRO (ethtool -k: tx-udp_tnl-segmentation, rx-udp_tnl-...), every encapsulated packet is handled segment-by-segment on the CPU → throughput can drop 3–5×. Modern NICs + kernels do this; verify on older ones.
Double conntrack, double qdisc, extra copies. Overlay ~5–15% throughput and a few µs latency vs routed mode, more if offloads are missing.
eBPF datapaths (Cilium) can bypass much of iptables/kube-proxy and parts of the bridge/veth path for a measurable win; eBPF host-routing and Direct Server Return for Services reduce hops further.
ipvs vs iptables kube-proxy: iptables mode is O(rules) per packet — thousands of Services → latency and reload spikes; IPVS or eBPF scale better.

56.5 Service VIPs, DNS, and NetworkPolicy — the TCP/IP view

A ClusterIP Service is a virtual IP with no host; kube-proxy/eBPF installs DNAT rules that rewrite VIP:port → a random ready pod IP, tracked in conntrack so replies un-DNAT. Load balancing is per-connection.
CoreDNS runs as pods behind a Service; every pod's /etc/resolv.conf points at the DNS ClusterIP with search <ns>.svc.cluster.local svc.cluster.local cluster.local and ndots:5 — see Ch. 20.8 for the query-amplification trap. NodeLocal DNSCache puts a caching resolver on each node to cut latency and conntrack churn.
NetworkPolicy is enforced by the CNI (Calico/Cilium) as stateful L3/L4 (and L7 with Cilium) filtering on the veth/eBPF path — default-allow until the first policy selects a pod, then default-deny for that pod.

56.6 War story

After a node-pool upgrade, cross-node pod traffic throughput fell from ~9 Gbit/s to ~1.5 Gbit/s; same-node was fine. The new node image lacked the NIC driver feature flag for VXLAN TSO/checksum offload, so the CPU segmented every encapsulated flow. ethtool -k eth0 | grep udp_tnl showed the offloads off [fixed]. Rolling to an image with the proper driver (or switching the CNI to routed/BGP mode, eliminating encap) restored line rate. Lesson: overlay throughput depends entirely on tunnel-aware NIC offloads; check tx-udp_tnl-segmentation after any kernel/driver/AMI change, or avoid encap with a routed CNI.

Try it yourself

ethtool -k eth0 | grep -E 'udp_tnl|gre|checksum|segmentation'   # tunnel offloads
ip -d link show flannel.1        # or vxlan.calico, cilium_vxlan - shows VNI, dstport, local
bridge fdb show dev flannel.1     # VTEP forwarding entries (pod MAC -> remote node IP)
sudo tcpdump -i eth0 -nn 'udp port 4789 or udp port 6081'    # VXLAN / GENEVE on the wire
sudo tcpdump -i eth0 -nn 'udp port 4789' -w /tmp/vx.pcap ; wireshark /tmp/vx.pcap  # decodes inner packet
cat /sys/class/net/$(ip route get 1.1.1.1 | awk '{print $5; exit}')/mtu
kubectl exec -it <pod> -- cat /etc/resolv.conf                 # ndots, search domains