Chapter 56 — TLS and certificate operations: expiry, chains, SNI, rotation
In one sentence
Most TLS outages aren't broken encryption. They're expired certificates, wrong names, missing intermediates, SNI mismatches, or clients trusting a different set of CAs than the one you tested with, and every one of them is an operational failure.
The problem
Chapter 29 explained how TLS works. In production, the hard part is the inventory: dozens or hundreds of certificates, on load balancers, CDNs, ingress controllers, internal services, and mobile apps, each with its own expiry date, renewal method, and owner. One forgotten certificate takes down the service that uses it, without warning, on the day it expires.
How it actually works: the chain
A server sends its leaf certificate plus the intermediates. The client builds a path from the leaf to a root it already trusts:
leaf: CN=example.com signed by -> intermediate: "Cloudflare TLS Issuing ECC CA 3"
intermediate signed by -> ... -> root in the CLIENT's trust store
Three facts explain most chain failures:
- The server must send the intermediates. Browsers sometimes fetch a
missing one themselves (AIA fetching) or have it cached, so "works in
Chrome" proves nothing.
curl, Java, Go, Python, and Android often don't fetch it, and they fail. - Trust stores differ. The OS, the browser, the JVM (
cacerts), Python'scertifi, and minimal containers (often no CA bundle) each have their own list of roots. - Roots expire too. On 30 September 2021 an old root that Let's Encrypt certificates chained to (DST Root CA X3) expired. Up-to-date clients switched to the newer root. Old Android versions, old OpenSSL, and embedded devices started failing on certificates that were themselves perfectly valid.
Certificate lifetimes are shrinking
In 2025 the CA/Browser Forum voted to cut the maximum lifetime of public TLS certificates in stages: 200 days from March 2026, 100 days from March 2027, and 47 days from March 2029. At 47 days, manual renewal stops being possible in practice. Every certificate needs automated issuance (ACME: Let's Encrypt, ZeroSSL, cloud certificate managers) and automated deployment to every place it's used.
Inspect a real endpoint
openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
| openssl x509 -noout -subject -issuer -dates -ext subjectAltName
curl -vI https://example.com/ 2>&1 | grep -E 'SSL|subject|expire|issuer'
Look for:
- certificate
Not Afterdate; - the Subject Alternative Name list containing the hostname (the CN is ignored by modern clients);
- issuer and chain: does
-showcertsprint the intermediates? - TLS version and cipher;
- whether SNI (
-servername) changes the certificate. Try with and without it.
Common fingerprints
| Symptom | Likely cause |
|---|---|
| Browser says expired | leaf cert expired, or the client's clock is wrong |
| Works in browser, fails in CLI/app | missing intermediate, or a different CA store |
| Works for some old devices, not others | root/cross-sign expiry; device trust store too old |
| Fails by IP, works by name | certificate names are DNS names; an IP isn't in the SAN |
| Wrong certificate behind load balancer | missing/wrong SNI routing, default certificate served |
| Handshake refused without a certificate | the edge has no certificate for that SNI at all |
| Mobile app fails after rotation | pinned cert/public key not updated |
| Internal service fails after "routine" CA rotation | clients still trust only the old private CA |
Build it in Go (30 min) — rotate a certificate with zero downtime
Restarting a server to load a new certificate drops connections. Go's
tls.Config.GetCertificate hook avoids that: it's called during every
handshake, so the server can switch certificates by swapping a pointer.
New connections get the new certificate, and existing connections finish on
the old one.
// certreload: rotate a TLS certificate with zero downtime (Chapter 56).
//
// The server never holds "the" certificate. tls.Config.GetCertificate is
// called during every handshake, so swapping an atomic pointer changes the
// certificate for all NEW connections while existing ones carry on untouched.
//
// go run ./certreload # generates two self-signed certs and rotates between them
package main
import (
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/tls"
"crypto/x509"
"crypto/x509/pkix"
"fmt"
"math/big"
"net"
"net/http"
"sync/atomic"
"time"
)
type certStore struct {
cur atomic.Pointer[tls.Certificate]
}
// GetCertificate runs inside every handshake. It can also choose by SNI
// (hello.ServerName) -- that is how one IP serves many certificates.
func (s *certStore) GetCertificate(hello *tls.ClientHelloInfo) (*tls.Certificate, error) {
return s.cur.Load(), nil
}
func main() {
store := &certStore{}
store.cur.Store(selfSigned("generation-1"))
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
panic(err)
}
srv := &http.Server{
Handler: http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.Write([]byte("ok\n")) }),
TLSConfig: &tls.Config{GetCertificate: store.GetCertificate, MinVersion: tls.VersionTLS12},
}
go srv.ServeTLS(ln, "", "") // empty paths: certificates come from GetCertificate
addr := ln.Addr().String()
fmt.Println("serving:", whoServes(addr))
// In production this would be triggered by a file watcher, SIGHUP, or a
// secret-manager callback after renewal. Load and VALIDATE first, then swap.
next := selfSigned("generation-2")
store.cur.Store(next)
fmt.Println("rotated. new connections now see:", whoServes(addr))
}
func whoServes(addr string) string {
// InsecureSkipVerify only because our demo certs are self-signed;
// never do this in real clients.
c, err := tls.Dial("tcp", addr, &tls.Config{InsecureSkipVerify: true})
if err != nil {
return err.Error()
}
defer c.Close()
leaf := c.ConnectionState().PeerCertificates[0]
return fmt.Sprintf("CN=%s serial=%s expires=%s", leaf.Subject.CommonName, leaf.SerialNumber, leaf.NotAfter.Format(time.DateOnly))
}
func selfSigned(cn string) *tls.Certificate {
key, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
serial, _ := rand.Int(rand.Reader, big.NewInt(1<<62))
tmpl := &x509.Certificate{
SerialNumber: serial,
Subject: pkix.Name{CommonName: cn},
DNSNames: []string{"localhost"},
NotBefore: time.Now().Add(-time.Hour),
NotAfter: time.Now().Add(90 * 24 * time.Hour),
KeyUsage: x509.KeyUsageDigitalSignature,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
}
der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key)
if err != nil {
panic(err)
}
return &tls.Certificate{Certificate: [][]byte{der}, PrivateKey: key}
}
$ go run ./certreload
serving: CN=generation-1 serial=2779963337554967105 expires=2027-01-01
rotated. new connections now see: CN=generation-2 serial=2047191123025242675 expires=2027-01-01
What to notice:
atomic.Pointermakes the swap safe while thousands of handshakes run concurrently, with no lock on the hot path.- Validate before you swap. In production, load the new pair with
tls.LoadX509KeyPair, parse the leaf, and check that the names, expiry, and key match. Only then store it. A bad certificate loaded hot is an instant outage. hello.ServerNameis the SNI. Serving many domains from one IP means a map lookup here, which is what Caddy, Traefik, and Kubernetes ingress controllers do.- Go's
golang.org/x/crypto/acme/autocertpackage plugs ACME into this same hook: it obtains and renews Let's Encrypt certificates automatically.
Exercises:
- Add a file watcher: reload when
cert.pem/key.pemchange on disk (or onSIGHUP), with validation and a log line showing old and new serial numbers. - Combine it with Chapter 29's
tlsinspect: runtlsinspect -expiry-days 30against every endpoint in a list. That list is your certificate inventory. - Do the Go plan's zero-downtime rotation (Day 54) and mTLS with identity extraction (Day 41) next.
Rotation rule
Certificate rotation is a deployment, not a file copy. Have:
- an inventory of every termination point (CDN, LB, ingress, app, mobile pins), the names on each certificate, and its owner;
- expiry monitoring at least 30 days out (14 when lifetimes reach 47 days), alerting from outside your network, on the certificate actually served;
- staging validation with old and new clients;
- a rollback path (keep the previous certificate and key);
- tests from outside and inside the network.
Common confusions
- "We use Let's Encrypt, so renewal is automatic." Issuance is automatic. Deployment to the load balancer, the CDN, and the Java keystore may not be. Monitor the certificate that's actually served, not the one in the directory.
- "Revoking a certificate stops it from working." Revocation checking is inconsistent. Browsers rely mostly on pushed revocation lists, and in 2025 Let's Encrypt stopped running OCSP altogether in favour of CRLs. Short lifetimes are the real revocation mechanism.
- "Certificate pinning makes us safer." It also gives you an outage whenever the pinned key changes. If you must pin, pin a CA's or intermediate's key, keep a backup pin, and have a remote kill switch.
Check yourself
- Why can a site work in a browser but fail from
curlor a Java service? - What does
GetCertificatelet a Go server do that loading a certificate at startup can't? - What will 47-day certificates force your team to change?
- List four places a certificate for
api.example.commight be deployed.
Further reading
- Tool: SSL Labs Server Test (ssllabs.com/ssltest) for chain, protocol, and client-compatibility analysis.
- Tool: crt.sh, a Certificate Transparency search. Find every certificate ever issued for your domains, including ones you didn't know about.
- Across the series: the HTTPS guide's certificate operations chapter covers ACME, CAA, and CT monitoring, and the Go guide's mTLS chapter builds client-certificate authentication end to end.