Chapter 55 — DNS operations: authoritative DNS, delegation, split-horizon, outages
In one sentence
Using DNS and operating DNS are different jobs. Operating it means delegation, TTL planning, authoritative servers, split views, DNSSEC, and changes made so that they can be rolled back.
The problem
DNS changes look trivial (edit one record) and fail in non-obvious ways. The
old answer stays in caches for hours, one resolver disagrees with another,
a typo in a delegation takes a whole domain offline, or a DNSSEC key rollover
turns every lookup into SERVFAIL. Several of the internet's biggest outages
were DNS outages.
Recursive vs authoritative
- Recursive resolver: answers clients by walking and caching the DNS tree.
Examples: your router, a corporate resolver,
1.1.1.1,8.8.8.8. - Authoritative server: owns a zone and publishes its records. It never asks anyone else.
Debug both sides separately:
dig www.example.com # via your configured resolver (cached)
dig +trace www.example.com # walk the tree yourself: root -> .com -> example.com
dig @8.8.8.8 www.example.com # a specific public resolver
dig @ns1.example.com www.example.com +norecurse # straight from the source, no cache
The flags: line tells you who answered: aa (authoritative answer) appears
only on answers that come straight from the authoritative server.
How it actually works: delegation, glue, and the SOA
A domain works because its parent points at its nameservers:
.com zone (run by the registry) example.com zone (run by you or your DNS provider)
───────────────────────────── ─────────────────────────────────────────────────
example.com. NS ns1.example.com. example.com. SOA ns1.example.com. hostmaster.example.com.
example.com. NS ns2.example.com. 2026100301 ; serial
ns1.example.com. A 192.0.2.53 <- GLUE 7200 ; refresh (secondaries poll)
ns2.example.com. A 198.51.100.53 <- GLUE 900 ; retry
1209600 ; expire
300 ; minimum = NEGATIVE-cache TTL
example.com. NS ns1.example.com.
www A 203.0.113.10 ; TTL 300
- Delegation = NS records in the parent zone. You change them at your registrar, not in your own zone file. If the parent's NS and your zone's NS disagree, resolvers behave unpredictably ("lame delegation").
- Glue = the A/AAAA records of nameservers that live inside the zone
they serve. Without glue, finding
ns1.example.comwould require askingns1.example.com. - SOA serial: secondaries copy the zone only when the serial increases.
Forgetting to bump it is a classic "my change isn't visible on half the
servers" bug. The
YYYYMMDDnnconvention makes it hard to forget. - Negative caching: an
NXDOMAINis cached too, for the SOA's minimum field or the SOA record's TTL, whichever is lower (RFC 2308). If you query a name before creating it, resolvers remember that it doesn't exist.
TTL is your rollback timer
If a record has a 1-hour TTL, clients and resolvers can keep the old answer for up to about an hour. A safe migration timeline:
T-48h lower TTL 3600 -> 60 (wait at least the OLD TTL for caches to pick it up)
T-0 change the record (most clients follow within ~60 s)
T+1h verify traffic moved; keep the old target running
T+24h raise TTL back to 3600 (cheaper, more resilient to DNS outages)
T+7d decommission the old target (some clients ignore TTLs: JVMs, embedded devices)
Long TTLs aren't bad: they make you resilient when your DNS provider is down, because cached answers keep working. Short TTLs make you agile. Choose per record.
Split-horizon DNS
The same name may intentionally resolve differently depending on where you ask from:
api.example.com from office VPN -> 10.10.5.20
api.example.com from internet -> 203.0.113.20
This is useful, but it creates "works for me" incidents. Always record which resolver answered. The same effect happens unintentionally with GeoDNS and CDNs, which return different addresses by location on purpose.
DNSSEC in one picture
DNSSEC adds signatures so a validating resolver can prove an answer wasn't forged:
root (trust anchor) --DS--> .com DNSKEY --DS--> example.com DNSKEY --RRSIG--> www A 203.0.113.10
Each parent publishes a DS record (a hash of the child's key). When the chain
breaks (an expired signature, or a key rollover where the DS at the
registrar wasn't updated), validating resolvers return SERVFAIL and the
domain disappears for every user of those resolvers. dig +dnssec shows the
ad (authenticated data) flag, and dig +cd (checking disabled) tells you
whether DNSSEC is the cause: if +cd works and a normal query fails, the
problem is the signature chain.
Build it in Go (25 min) — compare resolvers in one command
Most DNS incidents come down to "different resolvers give different answers". This program asks several resolvers at once, groups identical answers, and flags disagreement:
// dnsdiff: ask several resolvers the same question and show where the answers
// disagree -- split-horizon, stale caches, half-finished migrations, and
// "works for me" incidents (Chapter 55).
//
// go run ./dnsdiff www.github.com
// go run ./dnsdiff -servers system,1.1.1.1,8.8.8.8,9.9.9.9 api.example.com
package main
import (
"context"
"flag"
"fmt"
"net"
"os"
"slices"
"strings"
"sync"
"time"
)
func main() {
servers := flag.String("servers", "system,1.1.1.1,8.8.8.8,9.9.9.9", "comma-separated resolvers; 'system' = your OS config")
flag.Parse()
if flag.NArg() != 1 {
fmt.Println("usage: dnsdiff [-servers a,b,c] name")
os.Exit(2)
}
name := flag.Arg(0)
list := strings.Split(*servers, ",")
type answer struct {
addrs []string
took time.Duration
err error
}
answers := make([]answer, len(list))
var wg sync.WaitGroup
for i, s := range list {
wg.Add(1)
go func() {
defer wg.Done()
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
start := time.Now()
addrs, err := resolverFor(s).LookupHost(ctx, name)
slices.Sort(addrs)
answers[i] = answer{addrs, time.Since(start), err}
}()
}
wg.Wait()
groups := map[string][]string{} // identical answer sets -> which resolvers gave them
for i, a := range answers {
key := strings.Join(a.addrs, " ")
if a.err != nil {
key = "ERROR: " + a.err.Error()
}
groups[key] = append(groups[key], list[i])
fmt.Printf("%-10s %-8v %s\n", list[i], a.took.Round(time.Millisecond), key)
}
if len(groups) == 1 {
fmt.Println("\nall resolvers agree")
return
}
fmt.Printf("\n%d different answers. Possible reasons: split-horizon DNS, a record changed\n", len(groups))
fmt.Println("within the TTL (caches disagree), GeoDNS/CDN steering, or a broken resolver.")
}
// resolverFor returns a Go resolver that sends every query to one server.
// PreferGo uses Go's own DNS client, so the Dial hook decides the server.
func resolverFor(server string) *net.Resolver {
if server == "system" {
return net.DefaultResolver
}
return &net.Resolver{
PreferGo: true,
Dial: func(ctx context.Context, network, _ string) (net.Conn, error) {
var d net.Dialer
return d.DialContext(ctx, network, net.JoinHostPort(server, "53"))
},
}
}
$ go run ./dnsdiff www.github.com
system 94ms 20.207.73.82
1.1.1.1 81ms 20.207.73.82
8.8.8.8 80ms 20.207.73.82
9.9.9.9 81ms 20.205.243.166
2 different answers. Possible reasons: split-horizon DNS, a record changed
within the TTL (caches disagree), GeoDNS/CDN steering, or a broken resolver.
That disagreement is real and harmless: GitHub steers clients by location, and Quad9's resolver egresses from a different region. During a migration, the same output shows you exactly which caches still hold the old address.
What to notice: net.Resolver{PreferGo: true, Dial: ...} makes Go use its
own DNS client and send every query to the server you choose. The same
Dial hook is how you'd run DNS over TCP or through a SOCKS proxy.
Exercises:
- Add your company's internal resolver to
-serversand query an internal name from inside and outside the VPN. You've just mapped your split horizon. - Add an authoritative mode: look up the zone's NS records, then query each authoritative server directly. All of them should return the same SOA serial. If they don't, a secondary is behind.
- During your next DNS change, run it in a loop every 30 s and log when each resolver switches. Compare that with the TTL you set.
DNS incident checklist
- Does the domain delegate to the nameservers you think? (
dig NS example.com @a.gtld-servers.net) - Did you query the authoritative servers directly?
- Do all authoritative servers return the same SOA serial?
- Is DNSSEC involved? Does
+cdchange the result? - Did a CNAME point outside your control (a SaaS provider, a CDN)?
- Did an internal resolver cache a stale answer or a negative answer?
- Are clients using DoH/DoT and bypassing your intended resolver?
- Has the domain registration expired? It happens to large companies too.
Real-world outages
- October 2016, Dyn: a Mirai-botnet DDoS on a major DNS provider made dozens of large sites unreachable for hours. The lesson: use two independent DNS providers for critical domains.
- October 2021, Facebook: a routing change withdrew the BGP routes to
Facebook's own authoritative DNS servers. With no route to the nameservers,
every resolver in the world got
SERVFAILfor facebook.com, and internal tools that depended on the same DNS went down too. The lesson: don't make your recovery tools depend on the system that's down.
Common confusions
- "DNS propagation takes 48 hours." There's no propagation. There are caches, and they expire on the TTL you set. Plan the TTL and the change is predictable.
- "I can put a CNAME at the zone apex." The standard forbids a CNAME at
example.combecause it would conflict with the SOA and NS records there. Providers offer ALIAS/ANAME/"CNAME flattening" to work around it. - "SERVFAIL means the server is down." It means the resolver couldn't get a valid answer: upstream unreachable, a lame delegation, or a broken DNSSEC chain.
Check yourself
- Where do you change a domain's delegation, and why there?
- What are glue records, and when are they required?
- You created a record five minutes ago but a resolver still returns NXDOMAIN. Why, and how long until it doesn't?
- Write the TTL timeline for moving
api.example.comto a new load balancer. - How do you tell a DNSSEC failure from a nameserver outage?
Further reading
- Book: DNS and BIND (Liu & Albitz), still the best explanation of zones, delegation, and operations.
- Tool: DNSViz (dnsviz.net) visualises the delegation and DNSSEC chain for any domain and highlights what's broken.
- Across the series: the HTTPS guide's DNS chapter covers DNS from the browser's side, including the HTTPS/SVCB records that advertise HTTP/3, and Chapter 18 here builds a DNS client from raw bytes.