50. Anycast, ECMP, and BGP in one page

50.1 BGP — how the internet's routes are chosen

BGP (Border Gateway Protocol, RFC 4271) is the path-vector protocol that glues ~75 000 Autonomous Systems (ASes — networks under one administrative routing policy, each with an ASN) into one internet.

• Neighboring routers form a TCP session (port 179) and exchange route advertisements: prefix → AS_PATH (the list of ASes to traverse) plus attributes.
• A router picks the best path per prefix using a decision process roughly: highest LOCAL_PREF → shortest AS_PATH → lowest MED → eBGP over iBGP → lowest IGP cost to next hop → tie-breakers. Policy (LOCAL_PREF, filters) beats path length — the internet's shape is commercial, not shortest-path.
• eBGP between ASes; iBGP to distribute those routes inside an AS (needs full mesh or route reflectors).
• Relationships: customer (pays you, you announce them everywhere), peer (settlement-free, exchange only each other's customers), transit provider (you pay them for "the rest of the internet"). "Valley-free" routing.
• Convergence is slow (seconds to minutes) and there's no built-in authentication historically → route leaks and hijacks.

50.2 BGP incidents you should recognize

• Prefix hijack: an AS announces a prefix it doesn't own. Because longest- prefix match wins, announcing a /24 out of someone's /16 pulls that traffic globally (Pakistan/YouTube 2008; numerous crypto-DNS thefts).
• Route leak: an AS re-announces routes it shouldn't (e.g. transit routes to a peer), becoming an accidental transit and black-holing/degrading traffic (Level3 2017, others).
• Defenses: RPKI + ROV (Route Origin Validation — cryptographically asserts "AS X may originate prefix P"; routers drop ROA-invalid routes), IRR filtering, max-prefix limits, BGPsec (path validation, barely deployed), and MANRS best practices.

50.3 ECMP — Equal-Cost Multi-Path

When several next hops have equal cost, a router hashes each flow's 5-tuple to pick one — spreading load while keeping every packet of a flow on one path (so TCP doesn't see reordering). Consequences:

• A single TCP flow cannot exceed one path's capacity, even if 4 equal paths exist. Big transfers need multiple flows (or MPTCP).
• traceroute shows different IPs per probe at ECMP hops because each probe is a different flow tuple. Use Paris traceroute (keeps the tuple constant) to get a coherent path.
• A hash polarization bug (every router using the same hash) can send all traffic down one member. Good implementations seed the hash per device.
• Flowlet switching / adaptive load balancing (data-center fabrics) rebalance on idle gaps to avoid elephant-flow collisions.

50.4 Anycast — one address, many locations

The same IP prefix is announced via BGP from many sites. Each client's packets are routed by normal BGP to the topologically nearest announcement. Used by:

• Root/TLD DNS and public resolvers (1.1.1.1, 8.8.8.8).
• CDNs — an anycast VIP fronts hundreds of POPs; you hit the closest.
• DDoS scrubbing — spread a volumetric attack across global capacity.

Caveats:

• Stateless-friendly, stateful-fragile: UDP DNS is ideal. For TCP/TLS, a mid-connection BGP reconvergence can re-anchor the flow to a different site that has no state → RST. In practice reconvergence is rare enough, and each POP terminates TLS locally, so it works — but long-lived connections (websockets) are more exposed. Some designs use anycast only to find a POP, then hand off to a unicast address.
• "Nearest" is BGP-nearest, not geographically nearest — a peering quirk can send Lisbon to London instead of Madrid.

50.5 How a CDN request actually gets routed (tying it together)

1. DNS returns either an anycast VIP (Cloudflare, some Google) or a unicast IP chosen by the CDN's DNS based on the resolver's location / EDNS Client Subnet (Akamai-style).
2. Packets reach the nearest/edge POP via BGP (+ ECMP inside).
3. The POP's L4 LB (Maglev/Katran/XDP, consistent-hash) picks an edge proxy.
4. The edge proxy terminates TCP+TLS, serves from cache, or opens a connection over the CDN's private backbone (optimized routes, warm connections, BBR) to the origin.
5. Result: your 3-RTT TLS setup is against a server ~10 ms away, not the origin ~150 ms away; only cache misses pay the long haul, and even those ride a tuned backbone.

50.6 War story

A company's /24 for its API suddenly saw traffic vanish in three regions. A small ISP had fat-fingered a static route redistribution and originated their /24; with no RPKI ROA published, many networks accepted the bogus origin (equal prefix length, shorter AS_PATH from those networks' view). Mitigation: publish an RPKI ROA for the prefix, announce more-specific /25s temporarily to out-compete the hijack, and contact upstreams. Long-term: ROAs for every prefix + monitoring (BGPStream/Alerts). Lesson: if traffic disappears from some of the internet but not all, suspect a BGP hijack/leak; RPKI ROAs are the seatbelt.

Try it yourself

# Public looking glasses / RIPEstat:
curl -s "https://stat.ripe.net/data/routing-status/data.json?resource=93.184.216.0/24" | jq .
whois -h whois.radb.net -- '-i origin AS15169' | grep route | head   # prefixes an AS originates
mtr -z example.com          # -z shows the AS number per hop
traceroute -A example.com   # -A appends [ASxxxx]
# Is a prefix RPKI-valid? (Cloudflare's checker)
#   https://rpki.cloudflare.com/   or   `routinator`/`rpki-client` locally
dig +short CHAOS TXT id.server @1.1.1.1     # which anycast POP answered you