35. Notification Platform

Business context. As in §60.2: separate transactional notifications (OTP, receipts: critical, low latency) from engagement/marketing notifications (DAU uplift, but over-notifying causes permanent opt-outs).

Requirements. Multi-channel (push, SMS, email, in-app); templates and localization; user preferences and quiet hours; dedupe; scheduling; analytics (delivered, opened). Volume: 1B notifications/day, with marketing blasts of 100M in under an hour. Transactional p99 < 5 s end to end.

Design.

Producers (services, campaign tool) ─▶ Notification API (validate, idempotency key)
     ─▶ priority topics: [critical] [transactional] [marketing]   ← separate queues, quotas, workers
     ─▶ Processor: preferences & opt-outs ─▶ frequency caps/ranking (marketing only) ─▶ template render
     ─▶ Channel routers ─▶ provider adapters (APNs, FCM, SMS vendors A/B, email ESP) with per-provider
                           rate limits, circuit breakers, and failover between SMS vendors
     ─▶ Delivery status/webhooks ─▶ event log ─▶ analytics, retry, invalid-token cleanup

Deep dives.

• Priority isolation: a marketing blast must never delay OTPs. Use separate queues, worker pools, and provider quotas (bulkheads, §14).
• Dedupe: an idempotency key per (user, notification_type, entity) with a TTL.
• Provider limits and cost: SMS costs real money per message (cents, and more in some countries). Route by cost and deliverability, fall back to a secondary vendor, and prefer push or in-app when possible.
• Frequency capping and ranking: a budget per user per day across all teams that send notifications. Without a central budget, every team optimizes its own metric and the user uninstalls.
• Scheduling and time zones: "send at 9am local" means bucketing users by time zone and spreading the load.

Strategic lens. This is a platform that many teams depend on. Governance (who can send what, approval for large campaigns), self-serve templates, and per-team quotas and cost attribution matter as much as throughput.

🏗️ Flow: one notification from event to device, with all the checks.

 "order_shipped" event ─▶ Notification API (idempotency key = user:order:type)
   ─▶ [transactional] queue ─▶ Processor:
        prefs(user): email ✓ push ✓ sms ✗ ; quiet hours? (user TZ) → defer push to 08:00 local
        dedupe window (Redis SETNX key TTL 24h) ✓
        render template (locale de-DE, fallback en) ─▶ channel fan-out
   ─▶ Push router ─▶ APNs (token valid?) ─▶ 410 Unregistered → delete token
   ─▶ Email router ─▶ ESP A (circuit open?) → ESP B
   ─▶ delivery events (sent/delivered/opened/bounced) ─▶ Kafka ─▶ analytics + suppression lists

🌍 How it's built in production.

• LinkedIn "Air Traffic Controller" (ATC): a centralized service that decides whether, when, and through which channel a member gets a notification, applying frequency caps and relevance across all teams. It exists to prevent over-notification, a business outcome problem (§60.2).
• Uber RAMEN: the in-app real-time channel, separate from OS push. Use in-app channels for active users and OS push for inactive ones.
• Amazon SNS/SES, Twilio, SendGrid: most companies buy the last mile and build the policy layer (preferences, budgets, templates). That's a typical build-vs-buy answer.

⚠️ Corner cases.

• Notification about deleted content: the comment was deleted before the delayed push went out. Re-validate at send time.
• Time zones and DST for quiet hours and "9 a.m. local" campaigns. Store each user's IANA time zone, not a UTC offset.
• SMS deliverability by country: some countries require sender registration, and some routes silently drop messages. Fall back to voice OTP or another provider, and monitor delivery rate per country.
• Marketing to opted-out users is a legal violation (CAN-SPAM, GDPR, TCPA). Suppression lists must be checked at send time, with a strongly consistent read.
• Device token churn: reinstalls and new phones. Handle provider feedback (APNs 410, FCM UNREGISTERED) and prune tokens.
• Cross-channel duplicates: the user gets a push and an email for the same event while active in the app. Suppress based on in-app presence, with a short delay before external channels.