Frontend Labs
⋯

Chapter 56 — TLS and certificate operations: expiry, chains, SNI, rotation

In one sentence

Most TLS outages aren't broken encryption. They're expired certificates, wrong names, missing intermediates, SNI mismatches, or clients trusting a different set of CAs than the one you tested with, and every one of them is an operational failure.

The problem

Chapter 29 explained how TLS works. In production, the hard part is the inventory: dozens or hundreds of certificates, on load balancers, CDNs, ingress controllers, internal services, and mobile apps, each with its own expiry date, renewal method, and owner. One forgotten certificate takes down the service that uses it, without warning, on the day it expires.

How it actually works: the chain

A server sends its leaf certificate plus the intermediates. The client builds a path from the leaf to a root it already trusts:

 leaf: CN=example.com          signed by -> intermediate: "Cloudflare TLS Issuing ECC CA 3"
 intermediate                  signed by -> ... -> root in the CLIENT's trust store

Three facts explain most chain failures:

  1. The server must send the intermediates. Browsers sometimes fetch a missing one themselves (AIA fetching) or have it cached, so "works in Chrome" proves nothing. curl, Java, Go, Python, and Android often don't fetch it, and they fail.
  2. Trust stores differ. The OS, the browser, the JVM (cacerts), Python's certifi, and minimal containers (often no CA bundle) each have their own list of roots.
  3. Roots expire too. On 30 September 2021 an old root that Let's Encrypt certificates chained to (DST Root CA X3) expired. Up-to-date clients switched to the newer root. Old Android versions, old OpenSSL, and embedded devices started failing on certificates that were themselves perfectly valid.

Certificate lifetimes are shrinking

In 2025 the CA/Browser Forum voted to cut the maximum lifetime of public TLS certificates in stages: 200 days from March 2026, 100 days from March 2027, and 47 days from March 2029. At 47 days, manual renewal stops being possible in practice. Every certificate needs automated issuance (ACME: Let's Encrypt, ZeroSSL, cloud certificate managers) and automated deployment to every place it's used.

Inspect a real endpoint

openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName
curl -vI https://example.com/ 2>&1 | grep -E 'SSL|subject|expire|issuer'

Look for:

  • certificate Not After date;
  • the Subject Alternative Name list containing the hostname (the CN is ignored by modern clients);
  • issuer and chain: does -showcerts print the intermediates?
  • TLS version and cipher;
  • whether SNI (-servername) changes the certificate. Try with and without it.

Common fingerprints

Symptom Likely cause
Browser says expired leaf cert expired, or the client's clock is wrong
Works in browser, fails in CLI/app missing intermediate, or a different CA store
Works for some old devices, not others root/cross-sign expiry; device trust store too old
Fails by IP, works by name certificate names are DNS names; an IP isn't in the SAN
Wrong certificate behind load balancer missing/wrong SNI routing, default certificate served
Handshake refused without a certificate the edge has no certificate for that SNI at all
Mobile app fails after rotation pinned cert/public key not updated
Internal service fails after "routine" CA rotation clients still trust only the old private CA

Build it in Go (30 min) — rotate a certificate with zero downtime

Restarting a server to load a new certificate drops connections. Go's tls.Config.GetCertificate hook avoids that: it's called during every handshake, so the server can switch certificates by swapping a pointer. New connections get the new certificate, and existing connections finish on the old one.

// certreload: rotate a TLS certificate with zero downtime (Chapter 56).
//
// The server never holds "the" certificate. tls.Config.GetCertificate is
// called during every handshake, so swapping an atomic pointer changes the
// certificate for all NEW connections while existing ones carry on untouched.
//
//	go run ./certreload            # generates two self-signed certs and rotates between them
package main

import (
	"crypto/ecdsa"
	"crypto/elliptic"
	"crypto/rand"
	"crypto/tls"
	"crypto/x509"
	"crypto/x509/pkix"
	"fmt"
	"math/big"
	"net"
	"net/http"
	"sync/atomic"
	"time"
)

type certStore struct {
	cur atomic.Pointer[tls.Certificate]
}

// GetCertificate runs inside every handshake. It can also choose by SNI
// (hello.ServerName) -- that is how one IP serves many certificates.
func (s *certStore) GetCertificate(hello *tls.ClientHelloInfo) (*tls.Certificate, error) {
	return s.cur.Load(), nil
}

func main() {
	store := &certStore{}
	store.cur.Store(selfSigned("generation-1"))

	ln, err := net.Listen("tcp", "127.0.0.1:0")
	if err != nil {
		panic(err)
	}
	srv := &http.Server{
		Handler:   http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.Write([]byte("ok\n")) }),
		TLSConfig: &tls.Config{GetCertificate: store.GetCertificate, MinVersion: tls.VersionTLS12},
	}
	go srv.ServeTLS(ln, "", "") // empty paths: certificates come from GetCertificate
	addr := ln.Addr().String()

	fmt.Println("serving:", whoServes(addr))
	// In production this would be triggered by a file watcher, SIGHUP, or a
	// secret-manager callback after renewal. Load and VALIDATE first, then swap.
	next := selfSigned("generation-2")
	store.cur.Store(next)
	fmt.Println("rotated. new connections now see:", whoServes(addr))
}

func whoServes(addr string) string {
	// InsecureSkipVerify only because our demo certs are self-signed;
	// never do this in real clients.
	c, err := tls.Dial("tcp", addr, &tls.Config{InsecureSkipVerify: true})
	if err != nil {
		return err.Error()
	}
	defer c.Close()
	leaf := c.ConnectionState().PeerCertificates[0]
	return fmt.Sprintf("CN=%s serial=%s expires=%s", leaf.Subject.CommonName, leaf.SerialNumber, leaf.NotAfter.Format(time.DateOnly))
}

func selfSigned(cn string) *tls.Certificate {
	key, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
	serial, _ := rand.Int(rand.Reader, big.NewInt(1<<62))
	tmpl := &x509.Certificate{
		SerialNumber: serial,
		Subject:      pkix.Name{CommonName: cn},
		DNSNames:     []string{"localhost"},
		NotBefore:    time.Now().Add(-time.Hour),
		NotAfter:     time.Now().Add(90 * 24 * time.Hour),
		KeyUsage:     x509.KeyUsageDigitalSignature,
		ExtKeyUsage:  []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
	}
	der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key)
	if err != nil {
		panic(err)
	}
	return &tls.Certificate{Certificate: [][]byte{der}, PrivateKey: key}
}
$ go run ./certreload
serving: CN=generation-1 serial=2779963337554967105 expires=2027-01-01
rotated. new connections now see: CN=generation-2 serial=2047191123025242675 expires=2027-01-01

What to notice:

  • atomic.Pointer makes the swap safe while thousands of handshakes run concurrently, with no lock on the hot path.
  • Validate before you swap. In production, load the new pair with tls.LoadX509KeyPair, parse the leaf, and check that the names, expiry, and key match. Only then store it. A bad certificate loaded hot is an instant outage.
  • hello.ServerName is the SNI. Serving many domains from one IP means a map lookup here, which is what Caddy, Traefik, and Kubernetes ingress controllers do.
  • Go's golang.org/x/crypto/acme/autocert package plugs ACME into this same hook: it obtains and renews Let's Encrypt certificates automatically.

Exercises:

  1. Add a file watcher: reload when cert.pem/key.pem change on disk (or on SIGHUP), with validation and a log line showing old and new serial numbers.
  2. Combine it with Chapter 29's tlsinspect: run tlsinspect -expiry-days 30 against every endpoint in a list. That list is your certificate inventory.
  3. Do the Go plan's zero-downtime rotation (Day 54) and mTLS with identity extraction (Day 41) next.

Rotation rule

Certificate rotation is a deployment, not a file copy. Have:

  • an inventory of every termination point (CDN, LB, ingress, app, mobile pins), the names on each certificate, and its owner;
  • expiry monitoring at least 30 days out (14 when lifetimes reach 47 days), alerting from outside your network, on the certificate actually served;
  • staging validation with old and new clients;
  • a rollback path (keep the previous certificate and key);
  • tests from outside and inside the network.

Common confusions

  • "We use Let's Encrypt, so renewal is automatic." Issuance is automatic. Deployment to the load balancer, the CDN, and the Java keystore may not be. Monitor the certificate that's actually served, not the one in the directory.
  • "Revoking a certificate stops it from working." Revocation checking is inconsistent. Browsers rely mostly on pushed revocation lists, and in 2025 Let's Encrypt stopped running OCSP altogether in favour of CRLs. Short lifetimes are the real revocation mechanism.
  • "Certificate pinning makes us safer." It also gives you an outage whenever the pinned key changes. If you must pin, pin a CA's or intermediate's key, keep a backup pin, and have a remote kill switch.

Check yourself

  1. Why can a site work in a browser but fail from curl or a Java service?
  2. What does GetCertificate let a Go server do that loading a certificate at startup can't?
  3. What will 47-day certificates force your team to change?
  4. List four places a certificate for api.example.com might be deployed.

Further reading

  • Tool: SSL Labs Server Test (ssllabs.com/ssltest) for chain, protocol, and client-compatibility analysis.
  • Tool: crt.sh, a Certificate Transparency search. Find every certificate ever issued for your domains, including ones you didn't know about.
  • Across the series: the HTTPS guide's certificate operations chapter covers ACME, CAA, and CT monitoring, and the Go guide's mTLS chapter builds client-certificate authentication end to end.

Generated from real-life-guide-v1.md · 8 Oct 2026 16:36