Chapter 7 — NAT, VIP, BGP, Anycast & VRRP: Global Traffic Routing

How does your request find Netflix's servers across the world? And how does Netflix make sure a single server failure doesn't take down the site? These mechanisms answer both questions.

NAT — Network Address Translation

Covered in detail in Chapter 1 (How the Internet Works). Quick summary:

S-NAT (Source NAT) — Your Home Router

Your laptop         Router performs S-NAT        Netflix server
192.168.1.105:54321 ──────────────────────────► 52.38.36.82:443
                     rewrites source:
                     192.168.1.105:54321
                     → 203.0.113.45:54321

NAT table entry created:

Private                    Public                      External
192.168.1.105:54321   ←→  203.0.113.45:54321   ←→   52.38.36.82:443

Return traffic: Netflix responds to 203.0.113.45:54321. Router looks up NAT table. Delivers to 192.168.1.105:54321.

Why S-NAT exists: IPv4 address exhaustion. One public IP, many private devices.

D-NAT (Destination NAT) — Netflix's Load Balancer

Your request arrives at Netflix VIP:  Dst=52.38.36.82:443
Netflix LB rewrites destination:       Dst=10.0.4.55:443  (real backend)

This is D-NAT: rewriting the DESTINATION address.

Why D-NAT: One public VIP fans out to hundreds of private backend servers. This is the foundation of load balancing at L4.

PAT — Port Address Translation

A more specific form of S-NAT used by home routers. Multiple internal devices share ONE public IP — differentiated by different source ports.

Device A:  192.168.1.105:54321 → 203.0.113.45:54321  → Netflix
Device B:  192.168.1.107:54321 → 203.0.113.45:54322  → Netflix
Device C:  192.168.1.109:8080  → 203.0.113.45:54323  → Netflix

All three map to the SAME public IP but different source ports.
Router tracks all three in its NAT table.

VIP — Virtual IP Address

A VIP is an IP address that is not permanently assigned to any single server or physical interface. It is a floating address that can be held by different servers at different times.

Why VIPs Exist

Without a VIP:

You connect to real server IP: 10.0.4.55
Server crashes → your connection is dead
You must reconnect to a different IP: 10.0.4.56
You (and DNS) must know about multiple individual servers

With a VIP:

You connect to VIP: 52.38.36.82
VIP is held by whoever is healthy (load balancer or any server in the cluster)
One server crashes → VIP moves to another (VRRP) or load balancer removes it from pool
You stay connected to 52.38.36.82 — you never know which real server answered

Two Uses of VIP

1. External VIP — the public IP users connect to:

52.38.36.82  (advertised via BGP to the internet)
  → D-NAT → real servers inside (10.0.4.x range)

2. Internal VIP / Cluster VIP — inside the datacenter, for VRRP failover:

10.0.0.1  (internal VIP — held by VRRP master gateway)
  → If master fails, backup assumes 10.0.0.1
  → Internal servers still route through 10.0.0.1

BGP — Border Gateway Protocol

What BGP Is

BGP is the routing protocol of the internet — the system by which routers across the world exchange information about which IP address ranges they own and how to reach them.

Every company on the internet that owns IP addresses and runs its own routers has an Autonomous System Number (ASN):

Netflix:  AS2906
Google:   AS15169
Comcast:  AS7922
AT&T:     AS7018
Amazon:   AS16509

BGP lets these Autonomous Systems tell each other: "I own these IP prefixes. To reach them, send traffic to me."

BGP Route Announcement

Netflix (AS2906) announces to its BGP peers:
  "Prefix: 52.38.0.0/19 is reachable via AS2906"
  "Prefix: 198.38.96.0/19 is reachable via AS2906"

Your ISP (AS7922 — Comcast) receives this announcement.
Comcast's router adds to its BGP routing table:
  52.38.0.0/19  → next hop: AS2906 (Netflix)

When you request 52.38.36.82:
  Comcast router looks up 52.38.0.0/19 → forwards to Netflix

BGP Path Selection

Multiple paths to Netflix may exist. BGP selects the "best path" using a series of attributes (in order of preference):

1. Highest LOCAL_PREF     (prefer traffic leaving through preferred interface)
2. Shortest AS_PATH       (fewest hops through autonomous systems)
3. Lowest MED             (Multi-Exit Discriminator — server's preference)
4. eBGP over iBGP         (prefer external peers over internal)
5. Lowest IGP cost        (prefer closest next-hop internally)
6. Lowest Router ID       (tiebreaker)

Large networks use these attributes to engineer traffic — shifting load between ISP peers based on cost, capacity, policy, and performance.

BGP Convergence Time

When a BGP route is withdrawn (e.g., Netflix takes a POP offline), it takes 30–120 seconds for all routers worldwide to converge to the new routes. This is the fundamental speed limit of internet-level failover.

BGP Security — RPKI

BGP was designed without authentication. An AS can announce prefixes it doesn't own — this is called a BGP hijack. In 2010, Pakistan Telecom accidentally hijacked YouTube's IP space, taking YouTube offline globally.

RPKI (Resource Public Key Infrastructure): Allows IP address holders to publish cryptographically signed certificates (ROAs — Route Origin Authorizations) stating which ASes are authorized to announce their prefixes. Routers that validate RPKI will reject unauthorized announcements.

Anycast — One IP, Many Locations

What Anycast Is

Anycast is the practice of announcing the exact same IP prefix from multiple geographic locations simultaneously via BGP. BGP then routes each user to the nearest location.

Anycast is not a separate protocol. It is a strategy built entirely on top of BGP.

Normal Unicast (one IP → one location):
  52.38.36.82 exists only in Netflix US-East
  User in Tokyo → route to US-East → high latency → slow ✗

Anycast (one IP → many locations):
  52.38.36.82 announced via BGP from:
    POP in Tokyo      → peers with NTT Japan (2 AS hops from Tokyo users)
    POP in Frankfurt  → peers with DE-CIX (2 AS hops from Germany users)
    POP in Ashburn    → peers with Equinix (1 AS hop from East Coast users)
    POP in São Paulo  → peers with PTT-SP  (2 AS hops from Brazil users)

  User in Tokyo:
    BGP routing table: 52.38.36.82 → Tokyo POP (2 hops) ← BEST
                                   → Frankfurt (18 hops)
                                   → Ashburn   (14 hops)
    → Traffic goes to Tokyo POP ✓

  User in Frankfurt:
    BGP routing table: 52.38.36.82 → Frankfurt (2 hops) ← BEST
                                   → Ashburn   (8 hops)
                                   → Tokyo     (22 hops)
    → Traffic goes to Frankfurt POP ✓

  Same IP, different physical servers, BGP decides who answers.

Anycast Failover

Tokyo POP goes down (hardware failure, DDoS, maintenance):

  1. Netflix stops announcing 52.38.36.82 from Tokyo via BGP
     (BGP WITHDRAW message sent to all peers)

  2. Peer routers remove the Tokyo route from their tables

  3. BGP reconverges (~30–60 seconds globally)

  4. Tokyo users' BGP routing tables now show:
     52.38.36.82 → Osaka POP (next nearest, 4 hops)

  5. New connections go to Osaka POP ✓
     Existing connections (TCP) break — clients reconnect

  No DNS change needed. No client configuration change.
  Automatic, protocol-level failover.

BGP vs Anycast — The Relationship

BGP        = the mechanism (routing protocol that carries prefix announcements)
Anycast    = the strategy  (announce same prefix from many places)

BGP alone  → one prefix, one location (normal internet routing)
Anycast    → impossible without BGP (nothing to advertise the routes)
Anycast+BGP → same prefix from many locations, BGP routes to nearest

DNS Anycast vs HTTP Anycast

• DNS Anycast: 8.8.8.8 (Google DNS) is anycast — your query goes to the nearest Google DNS POP. Safe because DNS queries are stateless (UDP).

• HTTP Anycast: Trickier because TCP is stateful. If BGP reroutes mid-connection (e.g., POP fails), the TCP connection breaks. But new connections automatically go to the new nearest POP.

VRRP — Virtual Router Redundancy Protocol

The Problem VRRP Solves

Inside a datacenter, servers use a default gateway (usually the Top-of-Rack switch or a router) to send traffic out. If that gateway fails, all servers behind it lose connectivity — even if the internet is fine.

VRRP solves this with router redundancy: multiple physical routers share a single virtual IP address. If the master fails, a backup immediately takes over the VIP.

How VRRP Works

                    VIP: 10.0.0.1 (the default gateway all servers use)
                              │
              ┌───────────────┼───────────────┐
              │               │               │
        [Router A]      [Router B]      [Router C]
        MASTER ✓         BACKUP          BACKUP
        priority=150     priority=100    priority=50
        Owns VIP         Monitors A      Monitors A
              │
         [All servers use 10.0.0.1 as their default gateway]

VRRP heartbeats: The master sends VRRP advertisement multicast packets (224.0.0.18) every 1 second to the backup routers.

Failover sequence:

Router A (master) crashes:

  1. Router B and C stop receiving VRRP advertisements (wait 3 × interval = 3s)

  2. VRRP election: highest priority router becomes new master
     Router B wins (priority=100 > Router C priority=50)

  3. Router B sends GRATUITOUS ARP:
     "I am 10.0.0.1. My MAC is BB:BB:BB:BB:BB:BB"
     (Updates ARP caches on all local devices — they now send to Router B)

  4. Router B starts sending VRRP advertisements

  5. All servers behind 10.0.0.1 continue sending traffic — to Router B now ✓
     Failover complete in ~3 seconds (configurable)

VRRP vs VIP vs Anycast — Three Nested Layers

Level 1: GLOBAL (Anycast + BGP)
  Same prefix announced from multiple continents.
  BGP routes users to the right continent/POP.
  Failover: 30–60 seconds (BGP convergence)

Level 2: DATACENTER (VRRP)
  Multiple physical routers share a VIP inside one POP.
  VRRP routes traffic to the healthy router.
  Failover: 1–3 seconds (VRRP convergence)

Level 3: SERVER CLUSTER (Load Balancer / D-NAT)
  Multiple backend servers share a VIP via D-NAT.
  Load balancer routes requests to healthy servers.
  Failover: milliseconds (health check based)

These three layers work together. In a well-designed deployment, they route users toward an appropriate region, edge/router, and backend — with redundancy at each level.

Putting It All Together — Your Netflix Request

You request 52.38.36.82 (Netflix VIP):

1. Your router: S-NAT
   192.168.1.105:54321 → 203.0.113.45:54321

2. Your ISP's router: BGP routing table
   52.38.36.82 matches 52.38.0.0/19 → forward to Netflix AS2906

3. Netflix border router: BGP (Anycast)
   Multiple POPs announced this prefix.
   Nearest POP (e.g., Los Angeles) wins.
   Packet enters Netflix's LA POP.

4. VRRP at LA POP:
   VIP 52.38.36.82 is held by ToR Switch A (VRRP Master).
   Packet forwarded correctly.

5. Netflix L4 Load Balancer: D-NAT
   52.38.36.82:443 → 10.0.4.55:443 (real backend server)

6. Backend processes request, returns response.

7. Response travels back:
   D-NAT reversed: 10.0.4.55 → 52.38.36.82
   BGP routes back to your ISP
   S-NAT reversed at your router: 203.0.113.45 → 192.168.1.105
   Browser receives response.