12. Distributed Transactions: 2PC, Sagas, Outbox

12.1 Two-phase commit (2PC)

Coordinator: PREPARE → all participants vote YES/NO (and lock resources)
             all YES → COMMIT; any NO → ABORT

Gives atomicity across resources.
Blocking: if the coordinator dies after PREPARE, participants hold locks until it recovers.
Latency: at least 2 round trips plus forced log writes.
Use it inside one database or a NewSQL system (Spanner uses 2PC over Paxos groups, so the coordinator is itself replicated and the blocking problem goes away). Avoid it across independent microservices.

12.2 Sagas

A saga is a sequence of local transactions, each with a compensating action that undoes it semantically:

Book trip saga (orchestrated):
  1. ReserveFlight      ↔ compensate: CancelFlight
  2. ReserveHotel       ↔ compensate: CancelHotel
  3. ChargeCard         ↔ compensate: Refund
  4. ConfirmAll

  ChargeCard fails → run CancelHotel, CancelFlight (in reverse order)

Saga design rules:

Compensations must be idempotent and retryable forever (they can't fail permanently).
No isolation: other transactions see intermediate states. Use semantic locks (a status of PENDING), commutative updates, or reread-before-commit.
Put the step that is hardest to compensate last (the "pivot"), e.g. ship the order only after payment is captured.

12.3 TCC (Try-Confirm-Cancel)

A reservation-style variant: Try reserves resources (puts a hold on funds or inventory), Confirm finalizes, and Cancel releases. Holds have TTLs. This is the natural model for ticketing (§40) and payment authorizations (§39).

12.4 Decision table

Need	Use
Atomic multi-row change in one DB	A local ACID transaction
Atomic change across shards of one DB	The DB's distributed transactions (Spanner, CockroachDB, Vitess 2PC)
DB write + event publish	Transactional outbox / CDC
Multi-service business process	Saga (orchestrated with a workflow engine)
Reserve-then-commit resources	TCC with expiring holds

12.5 🏗️ Flow: orchestrated saga with a failure and compensations

 Orchestrator (Temporal/Step Functions)      Inventory       Payment        Shipping
   │ 1. ReserveStock(order 9) ───────────────▶ HOLD 2 units ✓
   │ 2. AuthorizePayment($80) ──────────────────────────────▶ AUTH ✓
   │ 3. CreateShipment ────────────────────────────────────────────────────▶ ✗ address invalid
   │    (retries exhausted / business error)
   │ C2. VoidAuthorization ─────────────────────────────────▶ VOID ✓ (idempotent)
   │ C1. ReleaseStock ───────────────────▶ RELEASE ✓ (idempotent)
   │ Order → FAILED, notify user
 Every step & compensation is recorded in the workflow history (durable),
 so an orchestrator crash resumes exactly where it stopped.

TCC states, including the two classic anomalies:

 Try ──▶ Confirm
   └──▶ Cancel
 "Empty rollback": Cancel arrives but Try never ran (Try was lost) → Cancel must succeed as a no-op
 "Hanging":        Try arrives AFTER Cancel (delayed network) → Try must be rejected,
                   or it reserves resources forever → record "cancelled" per txn id and check it in Try

12.6 🌍 In production

Uber Cadence → Temporal: Uber built Cadence (open source) for durable, long-running workflows (trips, payments, onboarding). Its creators later founded Temporal. Workflows are code, and their history is persisted so they survive process crashes and can run for days or months.
Netflix Conductor: an orchestration engine for content and encoding pipelines and other microservice workflows. Netflix chose orchestration because the choreographed flows had become impossible to follow.
Airbnb Orpheus: splits every payment workflow into a DAG of retryable idempotent steps, each with a pre-RPC, RPC, and post-RPC phase recorded in the database. Airbnb reports "five nines of consistency" for payments.
Google Spanner: runs 2PC across Paxos groups, so the coordinator and participants are each replicated. That removes 2PC's "coordinator died" blocking problem, which is why 2PC is acceptable inside Spanner but not across independent microservices.
Microsoft Durable Functions / Azure Logic Apps: the same orchestration pattern in serverless form. Name it in Microsoft interviews.

12.7 ⚠️ Corner cases & deep details

Compensation that can't fail, fails: the refund API is down for 6 hours. Compensations must retry with backoff indefinitely, then escalate to a human queue. The saga isn't done until every compensation succeeds.
Non-compensable actions: an email was sent, or a physical shipment left. You can only semantically compensate ("sorry, cancelled" email, return label). Put irreversible steps last (the pivot transaction).
Unknown outcome: a step times out. Did it happen? Query the downstream service for the status, or retry with the same idempotency key. Never assume it failed and compensate blindly, or you might refund a charge that never happened, or fail to refund one that did.
Lack of isolation: between "reserve stock" and "authorize payment", another saga reads the reserved stock as unavailable. Usually that's correct (a semantic lock), but sometimes it causes unfair rejections. Design the visible states deliberately.
Workflow versioning: changing workflow code while thousands of workflows are mid-flight. Temporal requires deterministic replay of history, so code changes must be gated by version markers, or old workflows break.
2PC in-doubt transactions: a participant voted YES and the coordinator vanished, so the participant holds locks indefinitely. Production 2PC (XA) needs coordinator recovery logs and operator tooling to resolve in-doubt transactions.