57. Cloud VPC constructs and their TCP/IP reality
57.1 The mental model
A cloud VPC is a software-defined L3 network. There is no real broadcast domain, no real switch you can see; the hypervisor's virtual switch + a distributed control plane emulate one. Almost every "weird" cloud-network behavior comes from that emulation.
57.2 Construct → what it actually is
| Cloud construct | TCP/IP reality |
|---|---|
| VPC / VNet | a routing domain + an IP prefix you chose (e.g. 10.0.0.0/16). Isolated from other VPCs unless peered. |
| Subnet | a slice of the VPC prefix pinned to one AZ. Not a broadcast domain — there's no ARP flooding; the platform answers ARP for the gateway and knows every instance's MAC/IP. The first ~3 IPs + broadcast are reserved by the provider (gateway, DNS, future use). |
| Route table | per-subnet. local route for the VPC CIDR is implicit and highest priority; you add routes to IGW / NAT GW / peering / TGW / VPN. Longest-prefix match as usual. |
| Internet Gateway (IGW) | 1:1 NAT between an instance's public IP and its private IP (the instance only ever sees its private IP; the mapping is stateless and done in the fabric). No bandwidth limit of its own. |
| NAT Gateway | managed PAT for private subnets → internet. Stateful, ~55k simultaneous connections per unique destination IP:port, idle timeout 350 s, charged per GB. Port exhaustion and the 350 s timeout are real app-visible limits (Ch. 21, 34). |
| Security Group | stateful allow-list on the ENI (Ch. 53.5). |
| Network ACL | stateless subnet filter — remember ephemeral return ports. |
| ENI (elastic network interface) | the instance's virtual NIC; carries the MAC, primary+secondary private IPs, SG bindings. Instance type caps # ENIs and IPs/ENI (→ pod density limits for VPC-CNI). |
| ELB/ALB/NLB | ALB = L7 proxy (terminates, new backend conn, adds X-Forwarded-For, HTTP/2 front). NLB = L4 (preserves client IP optionally, ~350 s idle, handles millions of flows, static IP per AZ). GWLB = transparent L3 for appliance insertion (GENEVE-encapsulates to firewalls). |
| VPC Peering | routes + fabric permission; non-transitive (A–B and B–C ≠ A–C). No encapsulation, MTU stays. |
| Transit Gateway / vWAN hub | a managed router connecting many VPCs/VPNs/Direct Connect; transitive; often caps MTU (e.g. 8500) and per-flow bandwidth (~5 Gbit/s per flow on AWS TGW — a single TCP flow can't go faster; use multiple flows). |
| PrivateLink / Private Service Connect | exposes one service across VPCs via an ENI in the consumer VPC + an NLB in the producer VPC; DNAT under the hood; no CIDR overlap concerns. |
| VPN (site-to-site) | IPsec tunnels; MTU ~1400 (do MSS clamping — Ch. 8); often ~1.25 Gbit/s per tunnel → use ECMP over multiple tunnels. |
| Direct Connect / ExpressRoute / Interconnect | private L2/L3 circuit to the cloud; supports jumbo (up to ~9001/8500); BGP for route exchange. |
57.3 MTU inside the cloud
| Path | Typical MTU |
|---|---|
| Within a VPC (same or peered), instance↔instance | 9001 (jumbo) on AWS/GCP |
| Over an Internet Gateway (to the internet) | 1500 |
| Over most VPN / some TGW attachments | 1300–8500 (varies; clamp MSS) |
| Cross-region VPC peering | often 1500 |
| Overlay CNI on top (Ch. 56) | subtract 50 (VXLAN) / 60–80 (encrypted) |
Set instance MTU and pod MTU to match the narrowest segment the traffic will cross, and allow ICMP "too big" through every SG/NACL.
57.4 Cloud DNS specifics
- Each VPC has a resolver at
VPC_base + 2(.2address) — a hidden recursive resolver that also serves private hosted zones and instance names. - Per-ENI DNS query throttle (~1024 packets/s to the
.2resolver on AWS). Chatty apps +ndots:5(Ch. 20.8) hit it → intermittentSERVFAIL/timeouts. Fix: local caching resolver / NodeLocal DNSCache, FQDNs, fewer lookups. - Split-horizon via private hosted zones; on-prem resolution via inbound/ outbound resolver endpoints.
57.5 Bandwidth & pps caps that surprise people
- Per-instance aggregate caps (e.g. "up to 10 Gbit/s" = burst, baseline
lower on small types) and per-flow caps (AWS: ~5 Gbit/s within a placement
group / same-AZ for a single flow, ~10 with ENA Express; less across AZ/region).
→ A single
scp/iperf3stream can't show you the instance's real capacity; use-Pparallel streams. - NAT GW / TGW / VPN per-flow limits as above.
- "Network burst credits" on small instances — great in tests, throttled under sustained load.
- PPS limits independent of bandwidth — small-packet workloads (DNS, RTP, trading) hit these first.
57.6 War story
A data pipeline moved TBs between two VPCs over a Transit Gateway. A single
transfer capped at ~5 Gbit/s despite 25 Gbit/s instances; the team suspected disk
or CPU. It was the TGW per-flow limit (~5 Gbit/s). Splitting the transfer
into 8 parallel TCP streams (s5cmd/rclone --transfers) aggregated to
~25 Gbit/s across the same TGW. Lesson: cloud networks impose per-flow
ceilings well below link speed at NAT GW / TGW / VPN / single-flow paths; design
bulk movers to use many parallel connections (Ch. 36).
Try it yourself
# Which resolver / MTU / route is in play on an instance:
resolvectl status 2>/dev/null || cat /etc/resolv.conf
ip route ; ip route get 8.8.8.8 ; cat /sys/class/net/eth0/mtu
ping -M do -s 8972 <peer-in-vpc> # jumbo within VPC?
ping -M do -s 1472 <internet-host> # 1500 to the internet
# Single vs parallel flow (reveals per-flow caps):
iperf3 -c <peer> -t20 ; iperf3 -c <peer> -t20 -P8
# NAT GW connection pressure (from instances behind it): watch for EADDRNOTAVAIL / timeouts
ss -tan state established dst <external-service> | wc -l
# AWS: instance-level network allowance exhaustion
ethtool -S eth0 | grep -E 'bw_in_allowance_exceeded|bw_out_allowance_exceeded|pps_allowance_exceeded|conntrack_allowance_exceeded'