28. Cost Engineering & Unit Economics
From Staff level up, and for every engineering manager, cost is a first-class requirement. Senior candidates who raise it unprompted stand out. Interviewers increasingly ask, "What does this cost, and how would you cut it by 30%?"
28.1 Unit economics
Define the cost of one unit of business value: cost per order, per active user, per GB stored, per 1k API calls, per 1M tokens. Then track it like latency.
Example: video platform
Storage: 1 PB at ~$20/TB-month (standard object storage) ≈ $20k/month
Egress: 50 PB/month delivered. At a blended CDN price of ~$0.005-0.02/GB → $250k-$1M/month
→ Egress dominates storage by 10-50x. Optimize codecs (AV1/HEVC: same quality,
~30-50% fewer bytes), cache hit ratio, and ISP-embedded caches (Netflix Open Connect)
before touching storage.
28.2 Common cost levers
| Lever | Typical saving | Trade-off |
|---|---|---|
| Right-sizing and autoscaling | 20–40% | Needs good metrics; scale-up lag |
| Committed use / reserved / savings plans | 30–60% on the baseline | Commitment risk |
| Spot / preemptible for batch, CI, stateless workers | 60–90% | Interruptions; needs checkpointing |
| Storage tiering (hot → infrequent → archive) | 50–90% on cold data | Retrieval latency and fees |
| Compression and better encodings (columnar, zstd, Protobuf) | 2–10× on storage and egress | CPU |
| Reduce cross-AZ/region traffic | Often surprisingly large | Locality-aware routing; replica placement |
| Caching | Fewer DB and compute instances | Staleness, cache-tier cost |
| Sampling telemetry | Large share of the observability bill | Less detail for rare events |
| Kill zombie resources and data retention | 5–20% | Requires ownership tags |
28.3 Build vs buy
| Factor | Favors buy (managed / SaaS) | Favors build |
|---|---|---|
| Differentiation | Commodity (auth, email, queues, observability) | It's your core product or moat |
| Scale | Small to medium; the vendor margin is less than your team cost | Huge; the vendor margin exceeds the cost of a dedicated team |
| Team | No expertise; on-call burden is unacceptable | Deep expertise already in house |
| Time to market | Need it this quarter | Can invest multiple quarters |
| Lock-in | Acceptable, or abstracted | Strategic risk (pricing power, roadmap) |
Total cost of ownership: a "free" open-source database costs engineers. At ~$250–400k fully loaded per engineer per year, a 3-person team to run it is about $1M per year before hardware.
28.4 🏗️ Architecture: where the dollar goes on one request
User request ($ per 1M requests, illustrative SaaS API)
│
├─ Edge/CDN + WAF ............... $0.5 (requests + egress)
├─ Load balancer ................ $0.3 (LCU/hour + data processed)
├─ Compute (app tier) ........... $4.0 ← right-size, autoscale, ARM instances, spot for async
├─ Cache ........................ $0.8
├─ Database ..................... $3.0 ← reserved capacity, read replicas vs cache
├─ Cross-AZ traffic ............. $1.2 ← HIDDEN: replication + chatty services across AZs
├─ NAT gateway processing ....... $0.9 ← HIDDEN: calls to cloud APIs/3rd parties via NAT
├─ Observability (logs/metrics) . $2.5 ← HIDDEN: often 10-30% of the infra bill
└─ Egress to internet ........... $1.8
Total ≈ $15 / 1M requests → at 3B requests/month ≈ $45k/month
Unit metric: $ per 1k active customers per month → tracked on the same dashboard as SLOs
28.5 🌍 In production
- Dropbox Magic Pocket: Dropbox moved most user data off Amazon S3 to its own exabyte-scale storage system (announced 2016). Its IPO filing reported infrastructure savings of roughly $75M over two years. Build beat buy because storage was the product, at extreme scale.
- Amazon Prime Video (2023): a monitoring service moved from serverless orchestration to a monolith on containers and reported about 90% lower cost for that workload. Per-invocation and state-transition pricing didn't fit high-frequency processing.
- Discord ScyllaDB migration: going from 177 to 72 nodes is a cost result as much as a latency result.
- Netflix Open Connect: caches inside ISPs cut transit costs for both Netflix and the ISPs, and moved the economics of streaming from paying for bandwidth to deploying hardware.
- 37signals' "cloud exit" (2023): moved steady, predictable workloads from the cloud to owned hardware, and publicly projected millions in savings over five years. It's the counter-example to "always cloud". Predictable, steady load with an ops team already in place changes the math.
28.6 ⚠️ Corner cases & deep details
- Cross-AZ data transfer is billed per GB in both directions on major clouds. A Kafka cluster with RF=3 across AZs, plus consumers in other AZs, can cost more in transfer than in brokers. Use rack-aware or zone-aware consumers (fetch from the nearest replica) and zone-aware routing.
- NAT gateway processing fees: private subnets pulling container images or calling S3 through NAT pay per GB processed. Use VPC endpoints.
- Observability cardinality and retention: unbounded labels and debug-level logs in production. Set per-team budgets and sample.
- Storage class minimums: archive tiers have minimum storage durations and retrieval fees. Moving short-lived data there costs more.
- Small-object request costs (§3.7): pack them into larger files.
- Commitments stranded by architecture changes: a 3-year reservation on an instance family, then you migrate to ARM or serverless. Prefer flexible savings plans, and coordinate the architecture roadmap with finance.
- Serverless break-even: per-request pricing wins for spiky, low-volume work and loses badly for sustained high-throughput work. Compute the crossover point for your workload.
- Idle multi-region: a warm standby region at 50% scale doubles a big part of the bill to protect an RPO/RTO the business may not need. Tie DR tiers to each service's business criticality (tier 0 gets multi-region; tier 2 gets backup and restore).