11. Asynchronous Processing: Queues, Logs, Streams

11.1 Message queue vs log

	Queue (SQS, RabbitMQ, Azure Service Bus, Pub/Sub)	Log (Kafka, Kinesis, Pulsar, Event Hubs)
Model	Messages are deleted when consumed and acknowledged	Append-only; consumers track offsets; data retained for days or forever
Replay	No (once acked, it's gone)	Yes: rewind the offset, add new consumers later
Ordering	Usually none (FIFO variants are limited)	Per partition
Fan-out	Needs a topic → queue per subscriber	Natural: each consumer group reads independently
Scaling consumers	Add workers freely	≤ 1 consumer per partition per group
Best for	Task/job distribution, work queues	Event streaming, CDC, event sourcing, analytics pipelines

11.2 Delivery semantics

At-most-once: ack before processing. Messages can be lost.
At-least-once: ack after processing. Messages can be duplicated. This is the practical default.
Exactly-once: achievable within a closed system (Kafka transactions spanning read-process-write within Kafka). Across a boundary to an external system it becomes effectively-once = at-least-once + idempotent consumer.

Idempotent consumers:

on message m:
  BEGIN
    INSERT INTO processed(message_id) VALUES (m.id)   -- unique constraint
    -- if conflict → already handled → ROLLBACK, ack, return
    apply side effects to DB
  COMMIT
  ack(m)

11.3 The dual-write problem and the transactional outbox

The bug: a service writes to its DB and then publishes to Kafka. If it crashes between the two, the DB and the downstream consumers disagree permanently.

The fix (outbox): write the event into an outbox table in the same DB transaction as the business change. A relay (a poller, or CDC such as Debezium) publishes the outbox rows to the broker. Delivery is at-least-once, so consumers must be idempotent.

 ┌─────────── one ACID transaction ───────────┐
 │ UPDATE orders SET status='PAID' ...         │
 │ INSERT INTO outbox(event_id, type, payload) │
 └──────────────────────────────────────────────┘
          │ CDC (Debezium) / poller
          ▼
       Kafka topic "orders" ──▶ inventory, email, analytics consumers (idempotent)

11.4 Ordering, partitioning, and backpressure

Ordering is only guaranteed per partition. Choose the partition key so that "things that must be ordered" share a key (order_id, account_id).
A poison message blocks an ordered partition. After N retries, move it to a DLQ (dead-letter queue), with alerting and a redrive tool.
Consumer lag is the key health metric. Alert on lag measured in time (seconds behind), not in message count.
Backpressure: bounded queues, producer rate limits, and load shedding. Unbounded queues turn a short overload into an hours-long latency incident, because everything in the queue is already too old to be useful.

11.5 Event-driven architecture: choreography vs orchestration

Choreography: services react to each other's events. Coupling is loose, but it's hard to see the end-to-end flow and hard to change it.
Orchestration: a workflow engine (Temporal, Step Functions, Durable Functions, Cadence) drives the steps. The flow is visible, retries and timeouts are built in, and long-running workflows are durable.

Rule of thumb: choreograph notifications of facts ("OrderPlaced") and orchestrate business processes with steps and compensations.

11.6 🏗️ Architecture: Kafka internals you'll be asked about

 Producer (idempotent: producer-id + per-partition sequence numbers; acks=all)
     │ key = order_id → partition = hash(key) % P
     ▼
 Topic "orders", P=12, replication factor 3, min.insync.replicas=2
   Partition 5:  Leader (broker 2)  ──replicate──▶ Follower (b4) ✓ in ISR
                                    ──replicate──▶ Follower (b7) ✗ lagging → removed from ISR
   Write is committed when all in-sync replicas (ISR) have it; with ISR={b2,b4} ≥ 2 → OK
   If ISR shrinks to 1 (< min.insync) → producer gets NotEnoughReplicas (writes fail, not lost)

 Consumer group "billing" (each partition → exactly one consumer in the group)
   C1: p0-p3   C2: p4-p7   C3: p8-p11
   commit offsets AFTER processing → at-least-once
   C2 dies / is too slow (max.poll.interval exceeded) → REBALANCE → p4-p7 move;
   uncommitted messages are re-delivered (duplicates!)

Outbox relay flow:

 Service txn: [UPDATE order] + [INSERT outbox row] ─commit─▶ Postgres WAL
 Debezium reads WAL ─▶ Kafka "order-events" (key=order_id) ─▶ consumers dedupe by event_id
 Outbox rows deleted/partition-dropped after relay confirms

11.7 Deeper: settings and semantics that decide correctness

Setting / concept	Correct default for important data	Why
`acks`	`all`	`acks=1` loses messages if the leader dies before followers copy them
`min.insync.replicas`	2 (with RF=3)	Guarantees at least 2 durable copies of every acked write
`unclean.leader.election.enable`	`false`	`true` lets an out-of-date replica become leader, which loses committed data in exchange for availability
Idempotent producer	on (default in modern clients)	Producer retries don't create duplicates or reorder within a partition
Transactions + `read_committed`	for read-process-write pipelines within Kafka	Atomic "consume offset + produce output" gives exactly-once within Kafka
Log compaction	for changelog / state topics	Keeps the latest value per key; tombstone (null value) deletes
Partition count	Size for peak consumer parallelism; hard to change	Adding partitions changes key → partition mapping, which breaks per-key ordering for in-flight keys
Rebalance protocol	cooperative-sticky / static membership	Avoids stop-the-world rebalances during deploys

11.8 🌍 In production

LinkedIn built Kafka (2011) to replace point-to-point pipelines with one durable, replayable log, so that every new consumer (search, analytics, monitoring) reads the same stream.
Uber runs one of the largest Kafka deployments (trillions of messages per day), with per-region clusters, aggregate clusters, and a uReplicator for cross-region replication. Its consumer proxy pushes messages to services over gRPC, which lets them scale past the one-consumer-per-partition limit.
Slack ("Scaling Slack's Job Queue", 2017) put Kafka in front of its Redis-based job queue. Redis had become a single point of failure under bursts, and Kafka gave durable buffering while Redis stayed as the execution layer.
Meta FOQS (Facebook Ordered Queueing Service): a distributed priority queue on sharded MySQL, used for async workloads that need priorities and delayed delivery, which Kafka doesn't provide natively.
Airbnb uses its Orpheus idempotency framework in event consumers to turn Kafka's at-least-once delivery into effectively-once processing.

11.9 ⚠️ Corner cases & deep details

Rebalance-induced duplicates: a consumer that takes longer than max.poll.interval.ms (a slow batch or a GC pause) is kicked out of the group. Its partitions move, and the new owner reprocesses everything since the last commit. Keep batches small, and make processing idempotent.
Retry topics break ordering: sending a failed message to retry-5m lets later messages for the same key get processed first. If ordering matters, block the key (park the whole key's stream) instead of the message.
DLQ redrive: replaying the DLQ days later applies old events over newer state. Consumers must compare versions or timestamps (an "only apply if newer" rule).
Replay side effects: rewinding offsets to rebuild a projection re-sends emails and re-charges cards unless side-effect consumers are separate from projection consumers, or dedupe by event ID forever.
Schema evolution: a producer adds a required field, and old consumers crash on deserialization. Enforce compatibility (BACKWARD/FULL) in a schema registry in CI.
Hot partitions: one key (a mega-tenant) saturates one partition and its consumer. Split by sub-key, at the cost of ordering only within the sub-key.
Outbox table growth: if the relay stalls, the outbox grows and slows the main database. Alert on outbox lag, and partition the outbox table by time.
Consumer lag ≠ health: lag can be 0 because the producer is broken. Alert on end-to-end freshness (now − event time of the last processed message) plus producer rate.