7. Caching

7.1 Caching patterns

7.2 Invalidation (one of the two hard things)

The safe default for cache-aside is update the DB, then delete the cache key (not "update the cache"):

Why delete instead of set?
  T1: read DB (v1) ......................... set cache=v1  ← stale!
  T2:            write DB (v2), set cache=v2
  If T1's set lands after T2's, the cache holds v1 indefinitely.
  Deleting means the worst case is one extra miss.

Remaining race (rare): T1 misses, reads v1; T2 writes v2, deletes key; T1 sets v1.
Mitigations: short TTL as backstop, versioned values (set only if newer),
leases (Facebook's memcache paper), or delete-after-delay ("double delete").

For stronger freshness, drive invalidation from the database's change stream (CDC → invalidate). This catches every writer, including batch jobs and admin scripts that never touch app code.

7.3 Failure modes and fixes

Single-flight in Go (collapse concurrent misses for the same key into one DB call):

import "golang.org/x/sync/singleflight"

var group singleflight.Group

func GetProduct(ctx context.Context, id string) (*Product, error) {
    if p, ok := cache.Get(id); ok {
        return p, nil
    }
    v, err, _ := group.Do(id, func() (any, error) {
        p, err := db.LoadProduct(ctx, id)
        if err != nil {
            return nil, err
        }
        cache.Set(id, p, ttlWithJitter(5*time.Minute))
        return p, nil
    })
    if err != nil {
        return nil, err
    }
    return v.(*Product), nil
}

7.4 Hit-ratio math (why the last 5% matters)

DB sees: total_QPS * (1 - hit_ratio)

100k QPS at 90% hit ratio → 10,000 QPS to DB
100k QPS at 99% hit ratio →  1,000 QPS to DB    (10x less!)
Drop from 99% to 95% during an incident → DB load 5x → likely outage.

This is why a cache that is "only an optimization" becomes load-bearing. You need capacity planning for cache-failure scenarios, not just for the steady state.

7.5 Where to cache (layers)

Browser/app cache → CDN edge → API gateway / reverse proxy → in-process (L1)
   → distributed cache (L2: Redis/Memcached) → DB buffer pool → disk

Each layer has a different invalidation story. A strong answer says which layers are used, what TTL each one has, and who is allowed to serve stale data, and for how long.

7.6 🏗️ Flow: cache-aside with leases and CDC-driven invalidation

 Read path                                         Write path
 ─────────                                         ──────────
 App ──GET k──▶ Cache                              App ──UPDATE──▶ DB (commit)
     ◀─miss + LEASE token L──                                       │ binlog / WAL
 App ──SELECT──▶ DB                                                 ▼
 App ──SET k v (lease L)──▶ Cache                       CDC consumer (Debezium / McSqueal-like)
        └─ rejected if k was deleted                               │
           after the lease was issued                              ▼
           (prevents stale set)                           Cache DELETE k  (all regions)
 Other readers during the miss:
   get "lease held, retry in 10ms" → wait (avoids thundering herd)

7.7 🌍 In production

• Facebook memcache ("Scaling Memcache at Facebook", NSDI 2013) is the reference design:
  • Leases solve both stale sets (a set with an invalidated lease token is rejected) and thundering herds (only one lease per key every few seconds; others wait or get slightly stale data).
  • McSqueal tails the MySQL commit log and broadcasts deletes, so invalidation doesn't depend on application code paths.
  • A gutter pool: when a memcache server fails, its requests go to a small idle pool with short TTLs instead of hammering the DB.
  • Cold cluster warm-up: a new cluster reads from a warm cluster on misses, not from the DB.
  • Regional pools and replication across frontend clusters.
• Discord puts request coalescing in its Rust data services: many concurrent requests for the same message become one DB query, which is what made a "hot channel" survivable.
• Instagram described caching promises rather than values: the first request stores an in-flight promise and concurrent requests await it. This is the same idea as single-flight, applied across a fleet.
• Netflix EVCache (Memcached-based) replicates cache data across AZs and regions, so losing a zone doesn't cause a miss storm against Cassandra.

7.8 ⚠️ Corner cases & deep details

• Set-before-commit bug: the code writes the cache inside a DB transaction that later rolls back. The cache now holds data that never existed. Invalidate after commit (an after-commit hook or CDC).
• Serialization version skew: during a rolling deploy, new pods write a new object format and old pods can't parse it, or the other way round. Version the cache key (user:v7:42) or keep formats backward compatible.
• Personalized data cached under a shared key: GET /me cached without the user ID in the key leaks one user's data to another. A classic security incident; it also applies to CDNs (see the CDN guide).
• Redis-specific traps:
  • Commands run single-threaded, so KEYS * or deleting a 10M-element set blocks the shard (use SCAN and UNLINK).
  • Replication is async, so a failover loses recent writes. Never use Redis as the only source of truth for money, locks that need correctness, or exactly-once counters.
  • Cluster mode: multi-key operations must be in the same hash slot (use {hashtags}); clients must handle MOVED and ASK during resharding.
• Eviction surprises: with maxmemory-policy noeviction, writes fail when memory is full. With allkeys-lru, important keys without TTLs get evicted in favor of junk. Decide explicitly.
• Caching errors: caching a transient DB error or empty result for 5 minutes turns a 1-second blip into a 5-minute outage. Use separate short TTLs for negative results and never cache 5xx responses.
• Big keys: one 50 MB value (a huge list) saturates the network for that shard and causes latency spikes for every key on it. Chunk or paginate big values.