21. Data Platforms: OLTP, OLAP, CDC & Stream Processing

21.1 Separate operational and analytical workloads

 OLTP services ──CDC──▶ Kafka ──┬──▶ Stream processing (Flink / Spark Streaming / Dataflow)
 (Postgres, Spanner,            │       └──▶ real-time features, alerts, materialized views
  DynamoDB)                     │
                                └──▶ Lake/Lakehouse (S3/GCS + Iceberg/Delta/Hudi)
                                        └──▶ Warehouse / OLAP (BigQuery, Snowflake,
                                             ClickHouse, Pinot) ──▶ BI, ML training

Never run analytics queries against the OLTP primary. It's the most common self-inflicted outage at growing companies.

21.2 Batch vs streaming: Lambda vs Kappa

Lambda: a batch layer (correct, slow) plus a speed layer (approximate, fast), merged at query time. Two codebases for the same logic.
Kappa: streaming only; reprocess by replaying the log. One codebase, but it needs long log retention and a capable stream engine.
In practice: streaming for freshness, plus periodic batch reconciliation for correctness on money-relevant numbers (billing, ad spend).

21.3 Stream-processing concepts you must know

Concept	Meaning
Event time vs processing time	When it happened vs when we saw it. Aggregate by event time.
Windows	Tumbling (fixed, non-overlapping), sliding/hopping (overlapping), session (gap-based)
Watermark	"We believe all events up to time T have arrived." Triggers window output.
Late data	Events after the watermark: drop them, send them to a side output, or update emitted results
State	Keyed state in a local store (RocksDB), checkpointed for recovery
Exactly-once	Checkpoints plus transactional or idempotent sinks

21.4 Data contracts and lineage (the Staff+ angle)

The biggest data-platform failures are organizational: an upstream team renames a field and silently breaks revenue reporting. Answers: schema registries with compatibility rules (Avro/Protobuf), data contracts owned by producers, lineage tracking, and data quality checks as CI gates.

21.5 🏗️ Flows: CDC handoff and event-time windows

CDC with initial snapshot (the tricky part):

 1. Record current WAL/binlog position P0
 2. Take consistent snapshot of table (as of P0) → emit as "read" events
 3. Stream changes from P0 onward → emit create/update/delete events
 Consumers must handle: snapshot rows + later updates for the same key (upsert by PK, keep newest LSN)
 Debezium-style "incremental snapshots" interleave chunks with streaming using watermarks to avoid
 a giant blocking snapshot.

Event time, watermarks, late data (1-minute tumbling windows):

 event times arriving:   12:00:10  12:00:40  12:01:05  12:00:55(late!)  12:01:30
 watermark (max_seen − 30 s allowed delay):
   after 12:01:30 seen → watermark 12:01:00 → window [12:00,12:01) FIRES with 3 events
   12:00:55 arrived before watermark passed → included ✓
   an event for 12:00:20 arriving at watermark 12:01:40 → LATE:
        drop | side-output for correction | update window (if allowed lateness) & re-emit

21.6 🌍 In production

Google Dataflow model (VLDB 2015) and MillWheel (2013) set out event time, watermarks, triggers, and accumulation modes, which became Apache Beam.
Google Photon (SIGMOD 2013) joins ad clicks with queries exactly once across data centers, with a Paxos-based registry of processed event IDs. This is the reference for §44.
Uber built Apache Hudi to do incremental upserts on its data lake, so the lake reflects database changes within minutes instead of daily full reloads.
LinkedIn Databus / Brooklin and Airbnb SpinalTap: in-house CDC systems built before Debezium matured. Each company concluded that the database log is the integration point.
Netflix Keystone: a Kafka + Flink platform processing trillions of events per day, offered self-serve to teams. The platform team owns the infrastructure; domain teams own their jobs.

21.7 ⚠️ Corner cases & deep details

Schema changes in CDC: a column rename upstream breaks every downstream consumer. Use a schema registry plus data contracts, and roll out expand/contract upstream (§29.3).
Deletes in a lake: append-only files don't support "delete user 42". Table formats (Iceberg, Delta, Hudi) add row-level deletes (merge-on-read or copy-on-write). GDPR forces this.
Small files: streaming writes thousands of tiny files per hour, which kills query performance and object-store request costs. Run scheduled compaction jobs.
Ordering across partitions: CDC orders events per key or partition, not across tables. Rebuilding a join of orders and order_items from two topics can see an item before its order. Buffer, or join on a transaction ID.
Backfills that double count: rerunning a day's job must overwrite that day's partition (idempotent output), never append.
Time zones in daily metrics: "daily revenue" in UTC vs local time disagree, and DST days have 23 or 25 hours. Define this in the data contract.
Watermark stalls: one idle Kafka partition holds the watermark back, so no window ever fires. Configure idleness detection.