Frontend Labs
⋯

Part 5 — Layer 3: Network

Job

Deliver a packet from any host to any other host across many networks, using hierarchical IP addresses and routing. L3 is "best effort": it can drop, duplicate, or reorder packets and doesn't tell anyone. Reliability is the job of a higher layer.

In our story

Earlier that evening, DHCP gave the laptop its L3 identity (the "DORA" exchange, over UDP 68 → 67):

Laptop → broadcast   DHCP DISCOVER  "I'm 3c:22:fb:aa:bb:01, I need an address"
Router → laptop      DHCP OFFER     "Try 192.168.1.23"
Laptop → broadcast   DHCP REQUEST   "I'll take 192.168.1.23"
Router → laptop      DHCP ACK       IP 192.168.1.23/24, gateway 192.168.1.1,
                                    DNS 192.168.1.1, lease 24h

At the click, the laptop's kernel makes a routing decision for 198.51.100.20 (the ALB's IP, which DNS supplied; see Part 9):

Is 198.51.100.20 inside my subnet 192.168.1.0/24?   No.
→ Use the default route: send it to the gateway 192.168.1.1
→ (L2) wrap it in a frame addressed to the gateway's MAC

The packet then travels about 12–18 router hops:

 hop  device                          what it does with our packet
 ───  ──────────────────────────────  ───────────────────────────────────────────────
  1   Home router 192.168.1.1         TTL 64→63, NAT: src 192.168.1.23:54012
                                         → 203.0.113.7:61001, forward to ISP
  2   ISP BNG                         TTL 63→62, route lookup, forward
 3–6  ISP aggregation/core            longest-prefix match on 198.51.100.0/24
                                         → "learned via BGP from the cloud's network"
 7–12 Transit / peering exchange      BGP-chosen path between autonomous systems
13–15 Cloud border + backbone         into the region, towards the ALB's subnet
 16   ALB                             destination reached; hand to L4

How it works

1. The IPv4 header (20 bytes, field by field):

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
┌───────┬───────┬───────────┬───┬───────────────────────────────┐
│Version│  IHL  │   DSCP    │ECN│         Total Length          │  4 | 5 | 0 | 0 | 1500
├───────┴───────┴───────────┴───┼─────┬─────────────────────────┤
│        Identification         │Flags│    Fragment Offset      │  0x1c46 | DF | 0
├───────────────┬───────────────┼─────┴─────────────────────────┤
│      TTL      │   Protocol    │        Header Checksum        │  64 | 6 (TCP) | ...
├───────────────┴───────────────┴───────────────────────────────┤
│                    Source IP: 192.168.1.23                    │
├───────────────────────────────────────────────────────────────┤
│                 Destination IP: 198.51.100.20                 │
└───────────────────────────────────────────────────────────────┘

The fields that matter in real life:

Field Why you care
TTL Decremented at every router. At 0 the packet is dropped and an ICMP "Time Exceeded" is sent back. That's what makes routing loops survivable, and it's how traceroute works. Default 64 (Linux/macOS), 128 (Windows)
Protocol 6 = TCP, 17 = UDP, 1 = ICMP. The hand-off to L4
DF (Don't Fragment) Set by modern stacks. A router that can't forward a too-big DF packet drops it and sends "ICMP Fragmentation Needed". Path MTU Discovery depends on that ICMP getting back to the sender
DSCP Quality-of-service marking. Respected inside enterprise/carrier networks, mostly ignored across the public Internet
Src/Dst The end-to-end identity, except where NAT rewrites it

2. Routing = longest-prefix match. Every router has a table of prefixes and picks the most specific one that matches:

Destination 198.51.100.20
  0.0.0.0/0         → upstream            (matches, /0)
  198.51.100.0/22   → peer A              (matches, /22)
  198.51.100.0/24   → peer B              (matches, /24)  ← WINNER: most specific

3. BGP: how the Internet agrees on routes. The Internet is about 75,000 Autonomous Systems (ASes: ISPs, clouds, large companies). Each one announces the prefixes it owns to its neighbours via BGP. Your ISP knows how to reach 198.51.100.0/24 because the cloud provider's AS announced it and that announcement spread. BGP chooses paths by policy and business relationships, not by shortest latency. That's why traffic sometimes takes odd detours.

4. NAT: the home router's trick. IPv4 has only about 4.3 billion addresses, so your whole home shares one public IP. The router keeps a translation table:

Inside (private)            Outside (public)          Remote
192.168.1.23:54012   ⇄   203.0.113.7:61001   ⇄   198.51.100.20:443
192.168.1.40:51877   ⇄   203.0.113.7:61002   ⇄   142.250.x.x:443   (your phone)

Consequences for software engineers:

  • NAT is stateful. Idle mappings expire (often after 30 s to 5 min for TCP), which silently breaks long-idle connections. That's why WebSockets and gRPC streams need keepalives.
  • Many mobile and some broadband users sit behind Carrier-Grade NAT (the 100.64.0.0/10 range): thousands of customers share one IP. Rate-limiting or blocking by IP punishes innocent users.
  • Servers can't initiate connections to clients behind NAT. That's why push notifications, WebRTC (STUN/TURN), and "phone home" agents exist.

5. ICMP: L3's messaging channel. ping (Echo), traceroute (Time Exceeded), and Path MTU Discovery (Fragmentation Needed) all depend on it. Blocking all ICMP is a classic misconfiguration that causes the MTU black-hole from Part 4.

6. IPv6 in one paragraph. 128-bit addresses, so every device can have a global address and no NAT is needed. A fixed 40-byte header, no fragmentation by routers, and ARP is replaced by Neighbour Discovery (ICMPv6). Browsers try IPv6 and IPv4 in parallel (Happy Eyeballs) and use whichever connects first. If your AAAA record points to a broken IPv6 path, some users get a 250 ms penalty on every new connection.

7. L3 in the cloud.

Cloud concept OSI reality
VPC 10.0.0.0/16 Your private L3 address space
Subnet 10.0.2.0/24 An L3 prefix, pinned to one availability zone
Route table Static routes: 0.0.0.0/0 → internet gateway or → NAT gateway
Internet gateway 1:1 NAT between a public IP and an instance's private IP
NAT gateway Many-to-one NAT (like your home router) for outbound traffic from private subnets
Security group Stateful L3/L4 firewall: allow tcp/443 from 0.0.0.0/0; return traffic is automatic
Network ACL Stateless L3/L4 filter per subnet: you must allow the return (ephemeral) ports too
VPC peering / Transit Gateway Routing between L3 address spaces. Requires non-overlapping CIDRs

Try it yourself

# My IP, subnet, gateway
ipconfig getifaddr en0; netstat -rn | grep default      # macOS
ip addr show; ip route show                              # Linux
ipconfig getpacket en0                                   # macOS: full DHCP lease

# Which route would the kernel use for this destination?
route -n get 198.51.100.20                               # macOS
ip route get 198.51.100.20                               # Linux

# My public IP (what the server sees after NAT)
curl -s https://ifconfig.me; echo

# The path, hop by hop (TTL trick). mtr = traceroute + live loss/latency stats
traceroute -n shop.example.com
mtr -rwc 50 shop.example.com

# Find the path MTU: largest packet that passes with DF set
ping -D -s 1472 shop.example.com      # macOS (-D sets DF)
ping -M do -s 1472 shop.example.com   # Linux. 1472 + 8 ICMP + 20 IP = 1500

What breaks

Symptom L3 cause
ping 192.168.1.1 works, ping 1.1.1.1 doesn't Default route missing, or upstream/ISP down
VPC peering request rejected / some services unreachable Overlapping CIDRs (both teams picked 10.0.0.0/16)
Connections hang only for some users / large payloads PMTU black-hole (ICMP blocked)
Traffic works one way, replies vanish Asymmetric routing through a stateful firewall that only sees half the flow
"Packets to region X take a crazy path" BGP policy; route leak; ISP peering dispute
Outbound calls from private subnet fail No route to the NAT gateway; NAT gateway in the wrong AZ/subnet
Rate limiter blocks a whole office/mobile carrier Many users behind one NAT/CGNAT IP

War story: "We can't connect the acquisition's network"

Company A bought Company B. Both had built their entire cloud estate in 10.0.0.0/16. VPC peering, Transit Gateway, and VPN all require non-overlapping address space. The fix took two quarters: re-IP'ing hundreds of services, redoing firewall rules and DNS, and running a NAT translation layer in the meantime. Lesson for managers: IP address planning is a company-wide, decade-long decision. Allocate CIDRs centrally (IPAM) from day one, even if you're "just a startup".

Why it matters to you

  • An IP address is not a user identity. NAT, CGNAT, proxies, and VPNs make sure of that. Design rate limits and fraud rules accordingly.
  • IP plans are architecture. Overlapping CIDRs are one of the most expensive network mistakes to fix.
  • Don't block ICMP wholesale. Allow at least "Fragmentation Needed" / "Packet Too Big".
  • Know which of your firewalls are stateful and which are stateless. Security groups and NACLs behave very differently, and teams lose days to that difference.
  • Long-lived idle connections die in NAT tables you don't control. Add application-level keepalives shorter than the smallest NAT timeout on the path.

Go deeper in the series


Generated from real-life-example-osi.md · 8 Oct 2026 16:36