Part XVIII — Cloud & container networking

55. Network namespaces, veth pairs, and bridges

55.1 The building block: the network namespace

A network namespace (netns) is an isolated copy of the entire network stack: its own interfaces, IP addresses, routing table, ARP/neighbor table, conntrack, iptables/nftables rules, and port space. A container is (mostly) a process in its own netns. localhost inside it is genuinely separate; two containers can both bind :8080.

ip netns add red
ip netns exec red ip addr           # only 'lo' (down) - a blank stack
ip netns exec red ip link set lo up

55.2 veth pairs — a virtual patch cable

A veth is a pair of interfaces; whatever enters one end exits the other. Put one end in the container's netns, leave the other in the host, and you have a link between them.

ip link add veth-red type veth peer name veth-red-host
ip link set veth-red netns red
ip -n red addr add 10.10.0.2/24 dev veth-red
ip -n red link set veth-red up
ip addr add 10.10.0.1/24 dev veth-red-host        # host end = the container's gateway
ip link set veth-red-host up
ip -n red route add default via 10.10.0.1
# NAT the container out to the internet:
sysctl -w net.ipv4.ip_forward=1
iptables -t nat -A POSTROUTING -s 10.10.0.0/24 -o eth0 -j MASQUERADE

That is essentially Docker's default bridge network, minus the bridge.

55.3 The Linux bridge — a software switch

To connect many containers on one host, put every container's host-side veth into a bridge (docker0, cni0, br0). The bridge does MAC learning and forwarding exactly like Ch. 7's switch, in software.

 container A netns        container B netns
   veth-a (10.10.0.2)       veth-b (10.10.0.3)
      |                        |
   veth-a-host             veth-b-host
      \_______  docker0  _______/     (bridge, 10.10.0.1, also the gateway)
                   |
                  eth0  --(MASQUERADE / SNAT)-->  physical network

Container↔container on the same host/bridge: pure L2, fast, no NAT.
Container→outside: default route to the bridge IP → host routes → SNAT to the host's IP.
Outside→container: DNAT (docker -p 8080:80 inserts an iptables/nftables DNAT rule hostIP:8080 → 10.10.0.2:80).
docker-proxy: a userspace fallback that also accept()s on the published port (handles hairpin/localhost cases); a source of confusion in captures.

55.4 What this costs / gotchas

Extra hops & MTU: each veth + bridge + NAT adds a little latency and (for overlays, Ch. 56) eats MTU — pod MTU is often 1450 or 1370. A pod MTU mismatch is the #1 cause of "TLS handshake hangs / big responses stall" in Kubernetes (same PMTU black-hole signature as Ch. 8).
conntrack pressure: every pod flow is a host conntrack entry; busy nodes need nf_conntrack_max raised (Ch. 53). kube-proxy in iptables/IPVS mode adds lots of rules/state.
SNAT port exhaustion: many pods sharing the node IP to one external service → ephemeral-port/tuple exhaustion (Ch. 21). Fixes: SNAT to a range of IPs, per-pod egress IP, or a dedicated egress gateway.
hairpin / NodePort quirks: pod reaching its own Service VIP, or NodePort + externalTrafficPolicy (Cluster SNATs and hides client IP; Local preserves client IP but only routes to pods on that node).
Bridge + iptables: br_netfilter / net.bridge.bridge-nf-call-iptables must be set for Kubernetes network policy to see bridged traffic.

55.5 War story

Pods on one Kubernetes cluster could curl small endpoints but every helm pull, image layer, and large API response hung. Node eth0 MTU was 9000 (jumbo), the CNI set pod veth MTU to 9000 too, but the cloud's VPC path capped at 8900 for cross-subnet traffic and dropped the DF packets while ICMP "too big" was filtered by a security group. Fixes: set pod/CNI MTU to 8900 (or 1450 to be safe), and allow ICMP type 3 / ICMPv6 type 2 in the security group. Lesson: in container networking, always verify the end-to-end pod MTU against the real underlay path; "small works, big hangs" = MTU.

Try it yourself

ip netns list
ip -n <ns> addr ; ip -n <ns> route ; ip -n <ns> neigh
bridge link ; bridge fdb show               # bridge ports + MAC table
nsenter -t <pid> -n ip addr                  # enter a container's netns by PID
ip link show type veth
# In a pod: check MTU and path
cat /sys/class/net/eth0/mtu
ping -M do -s 1400 <other-pod-ip>            # shrink until it passes = pod path MTU