5. APIs & Communication Protocols

5.1 Choosing the style

5.2 API design details interviewers probe

Pagination: use cursors, not offsets.

Offset:  GET /posts?offset=100000&limit=20
  → DB must scan and discard 100k rows; results shift when new rows are inserted.

Cursor:  GET /posts?after=eyJpZCI6IDk4NzY1fQ&limit=20     (opaque, base64 of {id: 98765})
  → WHERE id < 98765 ORDER BY id DESC LIMIT 20  → index seek, stable under inserts.

Idempotency for unsafe operations. Any POST that moves money, sends a message, or creates an order should accept an Idempotency-Key header, so the client can retry safely after a timeout:

POST /v1/payments
Idempotency-Key: 7f3c1e0a-...       (client-generated UUID)

Server:
  1. INSERT INTO idempotency(key, request_hash, status='in_progress')  -- unique on key
     - conflict & completed      → return stored response
     - conflict & in_progress    → 409 / retry-after
     - conflict & different hash → 422 (key reused for a different request)
  2. Execute the operation
  3. UPDATE idempotency SET status='done', response=... (same transaction as the effect, ideally)

Versioning. Use URL versions (/v1/) for public APIs and additive-only changes within a version. For internal APIs, use Protobuf field-number rules: never reuse a field number, only add optional fields.

Errors. Use structured errors with a stable machine-readable code, and mark which ones are retryable (429/503 with Retry-After).

Rate limits. Return 429 with limit/remaining/reset headers. See §32 for the design.

5.3 Sync vs async: the first big decision on every path

Synchronous (request/response)          Asynchronous (event/queue)
───────────────────────────────          ──────────────────────────
Caller needs the answer now               Caller only needs "accepted"
Simple to reason about, easy errors       Absorbs spikes, decouples deploys
Latency = sum of the chain                Latency = eventual; needs status API
Availability = product of the chain       Availability of producer ≈ queue's
Use: reads, auth, checkout pricing        Use: email, transcoding, fan-out,
                                               analytics, anything slow or flaky

Rule of thumb: only the work needed to give the user a correct answer belongs on the synchronous path. Everything else is an event.

5.4 🏗️ Flow: a timeout, a retry, and why idempotency saves you

 Client                      API server                         Payments DB / PSP
   │ POST /payments           │                                       │
   │ Idempotency-Key: K1      │                                       │
   │─────────────────────────▶│ INSERT idem(K1,'in_progress') ✓        │
   │                          │──── charge card ─────────────────────▶│ (succeeds)
   │   ...client timeout      │                                       │
   │   (3 s, network slow)    │◀─── OK ───────────────────────────────│
   │                          │ UPDATE idem(K1,'done', resp=201{...}) │
   │ retry POST /payments     │                                       │
   │ Idempotency-Key: K1      │                                       │
   │─────────────────────────▶│ SELECT idem(K1) → done                │
   │◀──── 201 {same body} ────│ (no second charge)                    │

Without K1, the retry charges the card twice. With K1 but without the in_progress state, two concurrent retries could both run the charge.

5.5 Deeper: protocol mechanics that matter in design

• Deadline propagation (gRPC): the client sets a deadline (e.g. 800 ms). Each hop passes the remaining time downstream. A service whose remaining time is below its expected latency should fail fast instead of doing work nobody will wait for.
• HTTP/2 head-of-line blocking: HTTP/2 multiplexes streams over one TCP connection, so one lost packet stalls every stream. HTTP/3 (QUIC) fixes this per stream. Uber moved its mobile push channel to gRPC over QUIC partly for lossy mobile networks.
• Protobuf compatibility rules: never change a field number or type; mark removed fields reserved; unknown enum values must be handled (old clients receiving new enum values); avoid required (proto3 removed it for this reason).
• GraphQL safety: limit query depth and complexity. GitHub's GraphQL API charges rate-limit points based on the number of nodes a query can return, not per request.
• Webhooks: sign payloads with HMAC plus a timestamp, reject old timestamps (replay window), deliver at least once with backoff for hours or days, expect out-of-order delivery, and give receivers an event ID for dedupe.

5.6 🌍 In production

• Stripe idempotency keys: described in Brandur Leach's posts on the Stripe blog. Keys are scoped per account, and responses are stored and replayed for retries. Stripe also pins each account to a dated API version and runs compatibility transforms, so old integrations never break ("APIs as infrastructure: future-proofing Stripe with versioning", 2017).
• Uber RAMEN (Real-time Asynchronous MEssaging Network): Uber's push channel to driver and rider apps. It started as Server-Sent Events with batched acks every ~30 s. Because a driver offer is valid for only ~30 s, unknown delivery state was unacceptable, so Uber moved to gRPC bidirectional streaming for instant acks ("Uber's Next Gen Push Platform on gRPC").
• Netflix moved its API aggregation layer to federated GraphQL so many domain teams can own parts of one client-facing graph.

5.7 ⚠️ Corner cases & deep details

• Idempotency-key edge cases: (1) two concurrent requests with the same key: the second must wait or get 409, never execute; (2) same key, different body: reject with 422; (3) the key expires (e.g. after 24 h) and a very late retry arrives: it would execute again, so document the window; (4) the operation partly succeeded, e.g. the charge went through but the DB update failed: you need a recovery point per step (Stripe's "atomic phases") so the retry resumes instead of restarting.
• Cursor pagination traps: sorting by a non-unique column (created_at) skips or duplicates rows on ties. Always add a tiebreaker: ORDER BY created_at DESC, id DESC with a cursor of (created_at, id). Rows whose sort key changes (ranked feeds) need a snapshot of the ranking per session, or users see duplicates.
• Timeouts don't cancel work: the client gave up, but the server keeps computing and writing. Propagate cancellation (context cancellation in Go, gRPC cancellation) and make writes idempotent.
• Mobile version skew: old app versions stay in use for years. Server APIs must stay compatible with clients you can't update. Use remote config or kill switches, and server-driven minimum versions as a last resort.
• Large payloads: a 50 MB JSON response blocks event loops and wastes memory. Stream it, paginate it, or return a presigned object-storage URL.
• Clock-based ordering in APIs: ?since=timestamp breaks with clock skew and same-millisecond writes. Use server-issued monotonic cursors or sequence numbers.