48. Walkthrough: Design Instagram

The prompt. "Design Instagram." Variants: "Design a photo-sharing service" (Google), "Design the Instagram feed" or "Design Instagram Stories" (Meta), "Design a social media app with images and a feed" (Microsoft and others). This is a breadth problem. Your main job is scoping, then going deep on two or three parts.

Posts:     100M posts/day           → ~1.2k/s avg, ~3.5k/s peak
Stories:   500M stories/day         → ~6k/s avg, ~18k/s peak
Feed reads: 500M DAU × ~10 feed fetches/day = 5B/day → ~60k/s avg, ~180k/s peak
Story tray reads: similar order     → ~60k/s avg
Likes:     4B/day                   → ~46k/s avg, ~140k/s peak (hot posts: thousands/s on ONE post)

Media storage (new per day):
  Photos: 100M × 70% × ~3 MB (original + 4-5 resized variants) ≈ 210 TB/day
  Videos: 30M × ~60 MB (all renditions)                          ≈ 1.8 PB/day
  Stories: 500M × ~1.5 MB avg (expire, but archived for "highlights"/memories) ≈ 750 TB/day
  → ~2.5-3 PB/day raw → ~1 EB/year → erasure coding (≈1.4×) + hot/warm tiering are mandatory

Egress: 5B feed fetches × 12 items × ~150 KB (adaptive-size images, video previews)
        ≈ 9 PB/day ≈ ~0.8 Tbps average just for feed images; video adds multiples
        → CDN + ISP-embedded caches; cost is dominated by delivery

Metadata: 100M posts × ~1 KB = 100 GB/day → ~36 TB/yr (+ indexes, replicas) → sharded SQL is fine
Feed cache: 500M DAU × 500 post IDs × 8 B ≈ 2 TB → a memory tier of tens of nodes (+ replicas)

POST /v1/media/upload_sessions {type, size, sha256}        → {media_id, upload_urls[] (resumable parts)}
POST /v1/posts  {media_ids[], caption, location?, audience} + Idempotency-Key → {post_id}
GET  /v1/feed?cursor=...                                     → {items[], next_cursor}
POST /v1/users/{id}/follow    DELETE /v1/users/{id}/follow
POST /v1/posts/{id}/likes     DELETE /v1/posts/{id}/likes    (idempotent: PUT semantics)
GET  /v1/posts/{id}/comments?cursor=...   POST /v1/posts/{id}/comments
POST /v1/stories {media_id}   GET /v1/stories/tray   POST /v1/stories/{id}/seen

 Mobile app ──▶ Edge (CDN for media; API edge for TLS + routing)
                   │
                   ▼
              API gateway (auth, rate limit) ──▶ GraphQL/BFF aggregation for mobile
   ┌──────────────┬───────────────┬─────────────┬──────────────┬────────────────┬─────────────┐
   ▼              ▼               ▼             ▼              ▼                ▼             ▼
 Upload/Media   Post svc       Graph svc     Feed svc      Engagement svc   Stories svc   Notification
 svc            (publish,      (follow,      (candidates,  (likes,          (tray,        svc
 (sessions,     visibility)    block, mute)  ranking,      comments,        seen state)
 processing)                                 hydration)    counters)
   │              │               │             │              │                │
   ▼              ▼               ▼             ▼              ▼                ▼
 Blob store     Sharded SQL    Graph store   Feed cache     Likes store     Stories KV (TTL)
 (hot→warm,     (posts,        + cache       (Redis/        + counter       + seen-state store
  erasure-coded) media meta)                 memcache)      aggregator
   │              │
   │              └─▶ outbox ─▶ Kafka "post_published" ──┬─▶ Fan-out workers ─▶ feed cache
   │                                                     ├─▶ Notification svc ─▶ APNs/FCM
   └─▶ Media pipeline (transcode, resize, moderation)    ├─▶ Search/Explore indexing
                                                         └─▶ Analytics / ML features
 Ranking svc (ML models, feature store) ◀── Feed svc, Explore svc

Deep dive 1: media storage and delivery (the cost center)

 Upload ─▶ Hot blob tier (SSD/HDD, 3× replication or fast EC)  ← first ~days: most reads happen here
        ─▶ age/popularity policy ─▶ Warm tier (erasure coded ~1.4×, HDD, fewer IOPS)
 Delivery: CDN (edge) ─miss─▶ regional cache ─miss─▶ origin "photo server" ─▶ blob store
 URL: https://cdn.example/v/t51/<media_id>_<variant>.jpg?sig=…&exp=…   (signed, expiring)

• Small-file problem: billions of small photos as separate files kill file-system metadata performance. Facebook's Haystack (OSDI 2010) packs photos into large append-only volume files with an in-memory index (one disk seek per read). f4 (OSDI 2014) moved older, "warm" blobs to erasure-coded storage, cutting the replication factor significantly.
• Variant strategy: precompute 4–5 common sizes at upload (instant delivery), or generate on the fly at the edge and cache them (less storage, more CPU, and risk on cache misses). Common answer: precompute the popular sizes, generate rare ones on demand.
• Formats: WebP/AVIF for images and adaptive bitrate for video (§37). A 20–30% byte reduction on egress is millions of dollars.
• Privacy at the CDN: a private account's media URL must not be shareable forever. Signed, expiring URLs, with the signature bound to the media ID and an expiry.

Deep dive 2: home feed generation and ranking

 Candidate generation (≈ 30 ms)                 Ranking (≈ 80-120 ms)               Post-processing (≈ 30 ms)
 ─────────────────────────────                  ─────────────────────               ─────────────────────────
 feed_cache[user] (pushed, ~500)                features: viewer×author affinity,   diversity (no 5 posts
 + celebrity posts I follow (pulled)            post age, media type, engagement    from one author in a row)
 + "unseen since last session" filter           velocity, viewer history            insert ads slots
 → ~500-1500 candidates                         multi-task model predicts           apply visibility filter
                                                P(like), P(comment), P(save),       hydrate top 20
                                                P(dwell>N s) → weighted score

• Hybrid fan-out: push for normal accounts, pull for accounts with more than ~100k followers (§33). Skip inactive followers.
• Ranking needs features fast: a feature store with online KV lookups, batch-fetched per request. Precompute author-level features; compute viewer-author affinity offline daily, with real-time updates for recent interactions.
• Latency budget: 300 ms p99 total, so the ranking model must run in about 100 ms for ~1,000 candidates. Use a cheap first-stage ranker to prune to ~200, then a heavy model.
• Why ranked, not chronological: Instagram moved to a ranked feed in 2016 because people were missing most posts from accounts they cared about. That's the business outcome behind the architecture.

Deep dive 3: likes and counters on hot posts

🧑‍💼 Interviewer: A celebrity's post gets 2 million likes in an hour. What happens?

🎤 Candidate: That's ~550 likes/s on one post, and peaks are much higher in the first minutes. Three separate concerns:

1. The like edge (post_id, user_id): a uniqueness-protected insert. It's idempotent, because a double tap is a no-op. Partitioned by post_id, the hot post is one partition, but at 550–5,000 writes/s a wide-column partition copes. Writes are small and append-like.
2. The counter: never UPDATE posts SET likes = likes + 1. Like events go to Kafka, and an aggregator sums them in memory per post and flushes every second to a sharded counter (N sub-counters per hot post, summed on read and cached). The display is approximate ("2.1M"), which is fine.
3. "Did I like it?" on every feed item: answer from the (user_id, post_id) index with a batch multi-get for the 20 posts in a page, cached per user.

Deep dive 4: Stories (ephemeral, high-churn)

• Why not reuse the feed? Stories expire after 24 h and are viewed as a tray of authors, not posts, and the tray order depends on "unseen first" plus affinity. Precomputing fan-out for 500M stories/day that expire is wasteful.
• Design: each author's active stories are a short list in a KV store with TTL. The tray is built at read time: take who I follow (~hundreds), keep those with active stories (a per-author "has active story" bitmap or cache), sort by unseen-first, then affinity. Cache the tray for a few minutes.
• Seen state: (viewer_id, author_id) → last_seen_story_ts is very high write volume (billions of views a day). Batch writes from the client (send seen events every few seconds), and store them in a write-optimized store with TTL ~48 h.
• "Seen by" list for authors: a celebrity story has 10M viewers, so store an approximate count plus the viewer list paginated, or capped for very large accounts.

Interviewer follow-ups & strong answers

🌍 Real-world references

• Instagram Engineering, "What Powers Instagram" (2011): Django, Postgres, Redis, Memcached, and a small team; "Sharding & IDs at Instagram" (2012): the ID scheme used in Step 4.
• The 2014 "Instagration": Instagram moved from AWS into Facebook's data centers, the classic "reuse the parent company's infrastructure" migration.
• "Open-sourcing a 10x reduction in Apache Cassandra tail latency" (2018): Instagram replaced Cassandra's storage engine with RocksDB ("Rocksandra") to remove JVM GC from the tail.
• "Dismissing Python Garbage Collection at Instagram" (2017): disabling GC improved memory sharing between worker processes (fewer copy-on-write page copies). A great "corner case at scale" story.
• "Powered by AI: Instagram's Explore recommender system" (2019): embedding-based candidate generation plus multi-stage ranking.
• Facebook Haystack (OSDI 2010) and f4 (OSDI 2014): photo storage at scale.

⚠️ Corner cases

• Carousel with 10 media, one fails processing: publish is atomic. The post goes live only when all media are ready. Otherwise the user gets a retry or remove choice.
• Retry after timeout creates duplicate posts: use an idempotency key per compose session (§5.2).
• EXIF location leaks home addresses: strip it at ingest. Location is only what the user explicitly tags.
• Story expiry: storage TTL cleanup lags behind exact expiry, so filter by expires_at at read time, never rely on deletion timing.
• Hot comment threads: a viral post gets 100k comments/hour. Paginate by (post_id, bucket), rank top comments separately, and rate-limit per user.
• Hashtags and Unicode: normalize (NFC, case folding) and handle emoji-only tags and right-to-left scripts.
• Account deletion (GDPR): a cascading async workflow across posts, media, likes, comments on others' posts, backups (crypto-shredding), search, and ML features, with an auditable completion record.
• Minors and safety: default-private accounts for teens and message restrictions. Policy requirements become architecture (age signals in visibility checks).

Signals by level on this problem

EM / Sr. EM: the Staff column plus ownership boundaries (media, feed and ranking, graph, integrity), a v1 cut, and on-call staffing for the media pipeline.