Chapter 28 — Resilience as a security property

In one sentence

Availability is the "A" in the CIA triad, so denial of service — whether from a flood, a retry storm, a noisy tenant, an algorithmic-complexity input, or a scaling bill — is a security concern, and rate limiting, quotas, timeouts, circuit breakers, bulkheads, and load shedding are security controls.

Where we are

You've secured confidentiality and integrity across the fleet. This chapter covers keeping it up under abuse, and the input classes that turn a single cheap request into an outage.

How it works

Rate limiting (throttle request rate):

   ALGORITHMS   token bucket (burst + steady rate), sliding-window
                counter/log, fixed window (simple, boundary spikes),
                leaky bucket.
   DIMENSIONS   per IP, per user, per TENANT, per API key, per endpoint,
                per (user x endpoint). Layer them -- a per-IP limit
                doesn't help against a distributed attack; a per-tenant
                limit does.
   WHERE        edge/CDN (volumetric), API gateway (per-key/route),
                a shared service (Envoy ratelimit, a Redis-backed
                limiter) for DISTRIBUTED limits across replicas, and a
                local in-process limiter as a cheap backstop.
   FAILURE MODE if the limiter (Redis) is down: fail-open (availability,
                but a DoS window) or fail-closed (protected, but you just
                DoS'd yourself)? Decide PER ENDPOINT -- login/OTP/password-
                reset lean toward a conservative local limit that works
                without the shared store; read APIs lean fail-open with an
                alert.
   RESPONSE     429 with `Retry-After`; don't do expensive work before
                rejecting; log for detection (credential stuffing shows up
                here, Ch 54).

Quotas (cap total consumption):

   Per tenant: requests/day, storage GB, compute-seconds, egress GB,
   emails/day, seats, API calls/month. Enforced at the gateway / a
   metering service.
   Purpose: a compromised or abusive tenant can't consume the whole
   platform's capacity OR run up an unbounded bill (DENIAL OF WALLET --
   the cloud-era DoS: the service stays up, the invoice doesn't).
   Pair with autoscaling MAX limits and cost-anomaly alerts.

Timeouts, retries, circuit breakers, bulkheads (stop cascading failure):

   TIMEOUTS       on every network call: connect, request, and per-
     dependency, propagated via context/deadline. No unbounded waits. A
     slow dependency must not pin all your worker threads.
   RETRIES        bounded (2-3), exponential backoff + JITTER, a RETRY
     BUDGET (cap retries as a fraction of total traffic -- prevents a
     retry storm amplifying an outage), only for idempotent ops / safe
     status codes, with an idempotency key.
   CIRCUIT BREAKER  after N consecutive failures to a dependency, OPEN the
     circuit (fail fast, don't call it) for a cool-down, then HALF-OPEN
     (probe). Stops you hammering a struggling service and turning its
     blip into your outage.
   BULKHEADS      separate resource pools (threads, connections) per
     dependency and/or per tenant, so one exhausting its pool doesn't
     starve the others.
   LOAD SHEDDING / BACKPRESSURE  when overloaded, reject early (429/503)
     and cheaply, ideally shedding low-priority traffic first (health
     checks and paying customers before batch/anonymous). Admission
     control at the front door.
   QUEUE-BASED LOAD LEVELLING  put a queue between spiky ingestion and
     capacity-bound processing; the queue absorbs bursts, workers pull at
     a sustainable rate (watch queue depth as a signal).

Algorithmic-complexity and resource-exhaustion inputs (cheap request, huge cost):

   ReDoS            catastrophic backtracking regex on attacker input
     (`(a+)+$`). -> use linear-time engines (RE2, Rust regex), timeouts,
     input length caps, review user-supplied or config regexes.
   Hash flooding    crafted keys colliding in a hash map -> O(n^2). -> use
     SipHash-keyed maps (most modern runtimes do), cap collection sizes.
   Decompression bombs  a 1KB zip/gzip/brotli that expands to 10GB;
     "billion laughs" XML entity expansion (Part 6 Ch 32). -> limits on
     decompressed size and ratio; disable entity expansion.
   Large/nested payloads  deep JSON, huge arrays, giant multipart, a 4MB+
     gRPC message. -> max body size, max depth, max fields/array length,
     streaming parsers.
   GraphQL           deeply nested / aliased / batched queries. -> query
     depth + complexity limits, cost analysis, persisted queries,
     disable batching or cap it (Part 6).
   Expensive-by-input  a search with wildcards, an unbounded date range, a
     report over all history, a pathological pagination. -> mandatory
     bounds, result caps, async for big jobs, per-query cost budgets.
   Amplification    one request fans out to many downstream calls / rows.
     -> cap fan-out; batch; paginate internally.

Resilience testing: chaos/fault injection (latency, errors, dependency kill) in staging; load tests that include abusive shapes; a "game day" that exercises the circuit breakers and load shedding for real.

Worked example

A single endpoint hardened against four different DoS vectors.

   POST /api/reports  { "type": "orders", "from": "...", "to": "...",
                        "filter": "<regex>" }

   Naive impl: builds a regex from `filter`, scans every order in the date
   range synchronously, returns the lot.

   Attacks:
     A) filter = "(x+x+)+y"  -> ReDoS, one request pins a CPU for minutes.
     B) from=2000-01-01 to=2030-01-01 -> scans all history, 20s, holds a
        DB connection + a worker thread.
     C) 500 concurrent such requests -> every worker thread and DB
        connection consumed -> the whole service is down (not just
        reports).
     D) a script hits it 10k times -> autoscaler spins up 200 pods ->
        $$$ (denial of wallet).

   Hardened:
     - `filter` is not a regex; it's a structured field set with an
       allowlist. (If regex is truly required: RE2, 50ms timeout, 200-char
       cap.)  -> A gone.
     - `from`/`to` required, max span 90 days, validated; older ranges ->
       "use the async export".  -> B bounded.
     - Reports run on a SEPARATE worker pool / service (bulkhead) with its
       own small DB connection pool; the main API is unaffected if reports
       saturate.  -> C contained to the reports subsystem.
     - Big reports are ASYNC: the endpoint enqueues a job, returns 202 +
       a job id; a worker processes at a sustainable rate; result to
       object storage with a short-lived presigned URL.  -> C, D bounded.
     - Rate limit: 5 report requests / minute / tenant, 60 / day quota;
       429 + Retry-After beyond that; alert on sustained 429s.  -> D, C.
     - Autoscaler max replicas capped; cost-anomaly alert on the reports
       service.  -> D visible and bounded.
   Now the worst case is "the reports feature is slow for one abusive
   tenant", not "SecureShop is down and the bill tripled".

Practice (60 min)

Add a token-bucket rate limiter (per user + per tenant + per endpoint) to a small API, backed by Redis for the distributed case and a local fallback. Load-test past the limit; verify 429 + Retry-After and that little work is done before rejecting.
Kill Redis; observe your fail-open vs fail-closed choice; implement a per-endpoint policy (login = conservative local limit; reads = fail-open + alert).
Add timeouts + bounded retries with jittered backoff + a retry budget to a downstream call. Simulate the downstream getting slow; confirm you don't exhaust threads and don't storm it.
Add a circuit breaker (e.g. resilience4j, opossum, gobreaker); simulate downstream failure; watch it open, cool down, half-open, close.
Add a ReDoS endpoint ((a+)+$ on user input); DoS it with one request; fix with RE2 / a timeout / an allowlist; re-test. Do the same with a zip bomb upload and a deep-JSON body — add size/ratio/depth limits.
Add per-tenant quotas and a cost/autoscale cap; simulate an abusive tenant and confirm blast radius is one tenant + a bounded bill.

Across the series

Resilience mechanisms built and measured in Go: rate limiting, idempotency (and the response-cache design that silently produced 52 orders from 40 requests), retry budgets, circuit breakers: HTTPS guide, Chapter 21. SLOs and burn-rate alerting: Chapter 23.
Slowloris and server timeouts: HTTPS guide, Chapter 19.

Common mistakes

Rate limiting only per IP. Useless against distributed abuse and shared NATs; add per-user and per-tenant.
The limiter as a SPOF. If a Redis outage takes down auth (fail-closed) or removes all protection (fail-open) with no alert, you've traded one outage for another. Choose per endpoint; have a local fallback.
Unbounded retries / no jitter / no budget. A downstream blip becomes a self-inflicted DDoS (thundering herd).
No timeouts. One slow dependency pins every worker; the whole service hangs.
Shared thread/connection pools across all dependencies and tenants. One bad actor starves everyone. Bulkhead.
Trusting user/config regexes, unbounded date ranges, deep payloads, big uploads. Each is a one-request outage. Cap everything.
No quotas / no autoscale max / no cost alerts. Denial of wallet: the service stays up while the invoice explodes.
Never testing the breakers. Resilience code that's never exercised doesn't work when you need it. Game days.

Check yourself

Why is denial of service a security concern, and which CIA property does it attack?
Name four dimensions to rate-limit on and why per-IP alone is insufficient.
What is a retry budget and what failure does it prevent?
Circuit breaker vs bulkhead — what does each contain?
Give four algorithmic-complexity / resource-exhaustion input classes and a mitigation for each.
What is "denial of wallet" and which controls bound it?