Frontend Labs
⋯

Chapter 55 — DNS operations: authoritative DNS, delegation, split-horizon, outages

In one sentence

Using DNS and operating DNS are different jobs. Operating it means delegation, TTL planning, authoritative servers, split views, DNSSEC, and changes made so that they can be rolled back.

The problem

DNS changes look trivial (edit one record) and fail in non-obvious ways. The old answer stays in caches for hours, one resolver disagrees with another, a typo in a delegation takes a whole domain offline, or a DNSSEC key rollover turns every lookup into SERVFAIL. Several of the internet's biggest outages were DNS outages.

Recursive vs authoritative

  • Recursive resolver: answers clients by walking and caching the DNS tree. Examples: your router, a corporate resolver, 1.1.1.1, 8.8.8.8.
  • Authoritative server: owns a zone and publishes its records. It never asks anyone else.

Debug both sides separately:

dig www.example.com                      # via your configured resolver (cached)
dig +trace www.example.com               # walk the tree yourself: root -> .com -> example.com
dig @8.8.8.8 www.example.com             # a specific public resolver
dig @ns1.example.com www.example.com +norecurse   # straight from the source, no cache

The flags: line tells you who answered: aa (authoritative answer) appears only on answers that come straight from the authoritative server.

How it actually works: delegation, glue, and the SOA

A domain works because its parent points at its nameservers:

 .com zone (run by the registry)          example.com zone (run by you or your DNS provider)
 ─────────────────────────────           ─────────────────────────────────────────────────
 example.com.  NS  ns1.example.com.       example.com.  SOA ns1.example.com. hostmaster.example.com.
 example.com.  NS  ns2.example.com.                         2026100301 ; serial
 ns1.example.com. A 192.0.2.53  <- GLUE                    7200       ; refresh (secondaries poll)
 ns2.example.com. A 198.51.100.53 <- GLUE                  900        ; retry
                                                            1209600    ; expire
                                                            300        ; minimum = NEGATIVE-cache TTL
                                          example.com.  NS  ns1.example.com.
                                          www           A   203.0.113.10   ; TTL 300
  • Delegation = NS records in the parent zone. You change them at your registrar, not in your own zone file. If the parent's NS and your zone's NS disagree, resolvers behave unpredictably ("lame delegation").
  • Glue = the A/AAAA records of nameservers that live inside the zone they serve. Without glue, finding ns1.example.com would require asking ns1.example.com.
  • SOA serial: secondaries copy the zone only when the serial increases. Forgetting to bump it is a classic "my change isn't visible on half the servers" bug. The YYYYMMDDnn convention makes it hard to forget.
  • Negative caching: an NXDOMAIN is cached too, for the SOA's minimum field or the SOA record's TTL, whichever is lower (RFC 2308). If you query a name before creating it, resolvers remember that it doesn't exist.

TTL is your rollback timer

If a record has a 1-hour TTL, clients and resolvers can keep the old answer for up to about an hour. A safe migration timeline:

 T-48h  lower TTL 3600 -> 60          (wait at least the OLD TTL for caches to pick it up)
 T-0    change the record              (most clients follow within ~60 s)
 T+1h   verify traffic moved; keep the old target running
 T+24h  raise TTL back to 3600         (cheaper, more resilient to DNS outages)
 T+7d   decommission the old target    (some clients ignore TTLs: JVMs, embedded devices)

Long TTLs aren't bad: they make you resilient when your DNS provider is down, because cached answers keep working. Short TTLs make you agile. Choose per record.

Split-horizon DNS

The same name may intentionally resolve differently depending on where you ask from:

api.example.com from office VPN  -> 10.10.5.20
api.example.com from internet    -> 203.0.113.20

This is useful, but it creates "works for me" incidents. Always record which resolver answered. The same effect happens unintentionally with GeoDNS and CDNs, which return different addresses by location on purpose.

DNSSEC in one picture

DNSSEC adds signatures so a validating resolver can prove an answer wasn't forged:

 root (trust anchor) --DS--> .com DNSKEY --DS--> example.com DNSKEY --RRSIG--> www A 203.0.113.10

Each parent publishes a DS record (a hash of the child's key). When the chain breaks (an expired signature, or a key rollover where the DS at the registrar wasn't updated), validating resolvers return SERVFAIL and the domain disappears for every user of those resolvers. dig +dnssec shows the ad (authenticated data) flag, and dig +cd (checking disabled) tells you whether DNSSEC is the cause: if +cd works and a normal query fails, the problem is the signature chain.

Build it in Go (25 min) — compare resolvers in one command

Most DNS incidents come down to "different resolvers give different answers". This program asks several resolvers at once, groups identical answers, and flags disagreement:

// dnsdiff: ask several resolvers the same question and show where the answers
// disagree -- split-horizon, stale caches, half-finished migrations, and
// "works for me" incidents (Chapter 55).
//
//	go run ./dnsdiff www.github.com
//	go run ./dnsdiff -servers system,1.1.1.1,8.8.8.8,9.9.9.9 api.example.com
package main

import (
	"context"
	"flag"
	"fmt"
	"net"
	"os"
	"slices"
	"strings"
	"sync"
	"time"
)

func main() {
	servers := flag.String("servers", "system,1.1.1.1,8.8.8.8,9.9.9.9", "comma-separated resolvers; 'system' = your OS config")
	flag.Parse()
	if flag.NArg() != 1 {
		fmt.Println("usage: dnsdiff [-servers a,b,c] name")
		os.Exit(2)
	}
	name := flag.Arg(0)
	list := strings.Split(*servers, ",")

	type answer struct {
		addrs []string
		took  time.Duration
		err   error
	}
	answers := make([]answer, len(list))
	var wg sync.WaitGroup
	for i, s := range list {
		wg.Add(1)
		go func() {
			defer wg.Done()
			ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
			defer cancel()
			start := time.Now()
			addrs, err := resolverFor(s).LookupHost(ctx, name)
			slices.Sort(addrs)
			answers[i] = answer{addrs, time.Since(start), err}
		}()
	}
	wg.Wait()

	groups := map[string][]string{} // identical answer sets -> which resolvers gave them
	for i, a := range answers {
		key := strings.Join(a.addrs, " ")
		if a.err != nil {
			key = "ERROR: " + a.err.Error()
		}
		groups[key] = append(groups[key], list[i])
		fmt.Printf("%-10s %-8v %s\n", list[i], a.took.Round(time.Millisecond), key)
	}
	if len(groups) == 1 {
		fmt.Println("\nall resolvers agree")
		return
	}
	fmt.Printf("\n%d different answers. Possible reasons: split-horizon DNS, a record changed\n", len(groups))
	fmt.Println("within the TTL (caches disagree), GeoDNS/CDN steering, or a broken resolver.")
}

// resolverFor returns a Go resolver that sends every query to one server.
// PreferGo uses Go's own DNS client, so the Dial hook decides the server.
func resolverFor(server string) *net.Resolver {
	if server == "system" {
		return net.DefaultResolver
	}
	return &net.Resolver{
		PreferGo: true,
		Dial: func(ctx context.Context, network, _ string) (net.Conn, error) {
			var d net.Dialer
			return d.DialContext(ctx, network, net.JoinHostPort(server, "53"))
		},
	}
}
$ go run ./dnsdiff www.github.com
system     94ms     20.207.73.82
1.1.1.1    81ms     20.207.73.82
8.8.8.8    80ms     20.207.73.82
9.9.9.9    81ms     20.205.243.166

2 different answers. Possible reasons: split-horizon DNS, a record changed
within the TTL (caches disagree), GeoDNS/CDN steering, or a broken resolver.

That disagreement is real and harmless: GitHub steers clients by location, and Quad9's resolver egresses from a different region. During a migration, the same output shows you exactly which caches still hold the old address.

What to notice: net.Resolver{PreferGo: true, Dial: ...} makes Go use its own DNS client and send every query to the server you choose. The same Dial hook is how you'd run DNS over TCP or through a SOCKS proxy.

Exercises:

  1. Add your company's internal resolver to -servers and query an internal name from inside and outside the VPN. You've just mapped your split horizon.
  2. Add an authoritative mode: look up the zone's NS records, then query each authoritative server directly. All of them should return the same SOA serial. If they don't, a secondary is behind.
  3. During your next DNS change, run it in a loop every 30 s and log when each resolver switches. Compare that with the TTL you set.

DNS incident checklist

  • Does the domain delegate to the nameservers you think? (dig NS example.com @a.gtld-servers.net)
  • Did you query the authoritative servers directly?
  • Do all authoritative servers return the same SOA serial?
  • Is DNSSEC involved? Does +cd change the result?
  • Did a CNAME point outside your control (a SaaS provider, a CDN)?
  • Did an internal resolver cache a stale answer or a negative answer?
  • Are clients using DoH/DoT and bypassing your intended resolver?
  • Has the domain registration expired? It happens to large companies too.

Real-world outages

  • October 2016, Dyn: a Mirai-botnet DDoS on a major DNS provider made dozens of large sites unreachable for hours. The lesson: use two independent DNS providers for critical domains.
  • October 2021, Facebook: a routing change withdrew the BGP routes to Facebook's own authoritative DNS servers. With no route to the nameservers, every resolver in the world got SERVFAIL for facebook.com, and internal tools that depended on the same DNS went down too. The lesson: don't make your recovery tools depend on the system that's down.

Common confusions

  • "DNS propagation takes 48 hours." There's no propagation. There are caches, and they expire on the TTL you set. Plan the TTL and the change is predictable.
  • "I can put a CNAME at the zone apex." The standard forbids a CNAME at example.com because it would conflict with the SOA and NS records there. Providers offer ALIAS/ANAME/"CNAME flattening" to work around it.
  • "SERVFAIL means the server is down." It means the resolver couldn't get a valid answer: upstream unreachable, a lame delegation, or a broken DNSSEC chain.

Check yourself

  1. Where do you change a domain's delegation, and why there?
  2. What are glue records, and when are they required?
  3. You created a record five minutes ago but a resolver still returns NXDOMAIN. Why, and how long until it doesn't?
  4. Write the TTL timeline for moving api.example.com to a new load balancer.
  5. How do you tell a DNSSEC failure from a nameserver outage?

Further reading

  • Book: DNS and BIND (Liu & Albitz), still the best explanation of zones, delegation, and operations.
  • Tool: DNSViz (dnsviz.net) visualises the delegation and DNSSEC chain for any domain and highlights what's broken.
  • Across the series: the HTTPS guide's DNS chapter covers DNS from the browser's side, including the HTTPS/SVCB records that advertise HTTP/3, and Chapter 18 here builds a DNS client from raw bytes.

Generated from real-life-guide-v1.md · 8 Oct 2026 16:36