14. Reliability Patterns & Availability Math

14.1 Availability composition

Serial (A calls B calls C, all required):
  A_total = A_a * A_b * A_c
  e.g. 5 services at 99.9% each → 0.999^5 ≈ 99.5%   (≈ 44 h/yr down!)

Parallel (redundant replicas, any one is enough):
  A_total = 1 - (1 - A)^n
  e.g. 2 replicas at 99%  → 1 - 0.01^2 = 99.99%
       (only if failures are independent — shared deploys, configs,
        and dependencies make them correlated)

Implications:

1. Deep synchronous call chains destroy availability. Prefer fewer hops, async work, and caching.
2. Redundancy only helps if failures are independent. A bad config pushed to all replicas at once makes your N replicas equivalent to 1. This is why staged rollouts and cells (§20) matter more than replica counts.

14.2 The resilience toolkit

Retry with full jitter in Go:

func callWithRetry(ctx context.Context, op func(context.Context) error) error {
    const maxAttempts = 4
    base, maxBackoff := 50*time.Millisecond, 2*time.Second
    var err error
    for attempt := 0; attempt < maxAttempts; attempt++ {
        if err = op(ctx); err == nil || !isRetryable(err) {
            return err
        }
        backoff := min(maxBackoff, base<<attempt)
        sleep := time.Duration(rand.Int63n(int64(backoff))) // full jitter: [0, backoff)
        select {
        case <-time.After(sleep):
        case <-ctx.Done(): // respect the caller's deadline
            return ctx.Err()
        }
    }
    return err
}

14.3 Failure domains

Name the failure domain of every component: process → host → rack → AZ → region → global (DNS, control plane, identity, config). Most large public outages trace back to the global layer, e.g. a bad config pushed everywhere or an expired certificate. Mature designs minimize global dependencies on the data path and roll out config like code (staged, with automatic rollback).

14.4 🏗️ Flows: deadlines, retry amplification, circuit breaker

Deadline propagation:

 Client deadline 1000 ms
  └▶ API (spends 50 ms) ──remaining 950──▶ Orders (spends 100) ──remaining 850──▶ Pricing
                                                                         └─ expected 900 ms > 850
                                                                            → fail fast (DEADLINE_EXCEEDED),
                                                                              don't start the work

Retry amplification (why one retry layer):

 Layer:      Edge   →   API   →   Service   →   DB
 Retries:     3         3          3            (attempts per layer)
 One DB failure under full retries = 3 × 3 × 3 = 27 DB calls per user request.
 With 5 layers: 3^5 = 243×. That alone can turn a blip into an outage.

Circuit breaker state machine:

          failures ≥ threshold in window
 CLOSED ───────────────────────────────▶ OPEN  (fail fast, return fallback)
   ▲                                       │ after cool-down (e.g. 30 s)
   │ probe succeeds                        ▼
   └──────────────────────────────── HALF-OPEN (let a few probe requests through)
                                           │ probe fails
                                           └──────▶ OPEN

14.5 🌍 In production

• Netflix Hystrix (2012) popularized circuit breakers and bulkheads (thread-pool isolation per dependency). It's now in maintenance mode, and Netflix moved toward adaptive concurrency limits ("Performance Under Load", 2018), which discover a service's concurrency limit automatically from latency changes, as TCP congestion control does, instead of using static thresholds.
• Netflix prioritized load shedding (2020): Zuul tags each request critical, degraded, or non-critical. Under stress it drops from the bottom. Playback requests survive while prefetch and logging calls are shed. Netflix reported that it prevented a major outage days after rollout.
• Stripe's four limiters ("Scaling your API with rate limiters", 2017):
  1. A request rate limiter.
  2. A concurrent-requests limiter.
  3. A fleet usage load shedder that always reserves capacity for critical methods.
  4. A worker utilization load shedder that sheds by traffic priority as workers saturate.
• Google SRE, "Handling Overload": client-side adaptive throttling. Each client tracks requests and accepts and rejects locally with probability max(0, (requests − K·accepts) / (requests + 1)), with K≈2. The backend stops wasting work on rejections it would have sent anyway.
• Amazon Builders' Library, "Avoiding fallback in distributed systems": fallback code paths are rarely exercised, so they fail when needed. Prefer making the primary path more reliable, or constantly exercise the fallback path.
• Netflix Chaos Monkey (2011) and the Simian Army: kill instances in production during business hours so resilience is tested continuously, not assumed.

14.6 ⚠️ Corner cases & deep details

• Timeouts longer than the caller's: if the caller times out at 1 s and the callee at 2 s, the callee does work nobody waits for. Timeouts must shrink going downstream (use deadlines).
• Retrying non-idempotent calls: retrying POST /transfer after a timeout double-sends. Only retry if the operation is idempotent or keyed.
• Breaker granularity: one breaker per service trips everything when only one bad host is failing. Use per-host outlier ejection plus a per-service breaker.
• Half-open herd: every client's breaker goes half-open at the same moment after the cool-down, so the recovering service gets hit at once. Add jitter to the cool-down.
• Shedding too late: rejecting a request after auth, DB reads, and serialization saves nothing. Shed at the edge, before expensive work.
• Queue + deadline: a request that waited 2 s in a queue for a 1 s deadline should be dropped when dequeued, not processed. Under overload, LIFO or CoDel-style queueing serves requests that can still succeed.
• Bimodal latency: a cache hit takes 1 ms, a miss 200 ms. Timeouts tuned to the hit case cause mass failures when caches are cold. Tune to realistic cold-path latency.
• Health checks hiding partial failure: the instance returns 200 on /health but fails every request to one shard ("gray failure"). Use request-level outlier detection, not only health endpoints.