17. Security & Multi-Tenancy in Design

The security guide covers the mechanics. In a design interview you need to place these correctly on the diagram:

Concern	Design-level answer
AuthN (users)	OIDC/OAuth 2.0 at the edge; short-lived access tokens plus refresh tokens; SSO/SAML for enterprise
AuthN (services)	mTLS with workload identity (SPIFFE, a cloud IAM role), not shared API keys
AuthZ	Central policy (RBAC/ABAC/ReBAC); a Google Zanzibar-style relationship store for sharing models such as Docs and Drive; enforced at every service
JWT vs opaque tokens	JWTs are self-contained (no lookup, hard to revoke, so keep them short-lived); opaque tokens need introspection (easy revocation)
Data protection	TLS everywhere; encryption at rest with KMS; envelope encryption; per-tenant keys for enterprise customers (and crypto-shredding for erasure)
Secrets	A vault or KMS, rotated, never in code or images
Abuse	Rate limits (§32), bot detection, WAF, quotas per API key
Auditing	Immutable audit log of admin and data access (often an enterprise sales requirement)
Privacy	Data classification, PII minimization, deletion pipelines (GDPR/CCPA), residency

Multi-tenancy models

Model	Isolation	Cost efficiency	Used for
Shared everything (a `tenant_id` column)	Logical only; noisy neighbors	Best	SMB/self-serve tiers
Shared compute, separate schema/DB	Data isolation	Medium	Mid-market
Cell per tenant group	Blast radius contained	Medium	Large SaaS platforms
Dedicated stack (single-tenant)	Full	Worst	Regulated or very large enterprise ("bring your own key", sovereign cloud)

The usual strong answer is tiered tenancy: pooled by default, with the ability to promote "whale" or regulated tenants to dedicated cells. Routing is done through a tenant directory. This maps directly onto pricing tiers, which is the business tie-in.

17.1 🏗️ Flow: authentication and authorization end to end

 User ─▶ App ─(OIDC auth code + PKCE)─▶ Identity Provider (Entra ID / Google / Okta)
                ◀── ID token + access token (JWT, 5-15 min) + refresh token (rotating) ──
 App ─▶ API Gateway: verify JWT signature (cached JWKS), expiry, audience, tenant claim
           │ forwards identity as signed internal header / token exchange (narrower scope)
           ▼
       Service A ──mTLS (SPIFFE workload identity)──▶ Service B
           │
           └─▶ AuthZ check: "can user:alice VIEW doc:123?"
                  ▼
               Policy/relationship service (Zanzibar-style):
                 doc:123#viewer@group:eng#member  and  user:alice ∈ group:eng → ALLOW
                 (answer evaluated at a consistency token ≥ the doc's last ACL change)

17.2 Deeper: authorization at scale (Zanzibar)

Google's Zanzibar (USENIX ATC 2019) is the authorization system behind Drive, YouTube, Calendar, Cloud, and others:

Stores relation tuples: object#relation@user (where the user can itself be a set such as group:eng#member).
Namespace configs define rewrites ("editors are also viewers", "viewers of a folder are viewers of its docs").
Built on Spanner. It handles millions of checks per second with low latency through aggressive caching and request hedging.
The "new enemy" problem: Alice removes Bob from a doc, then adds secret content. If the ACL check reads a stale replica, Bob sees the new content. Zanzibar solves it with zookies: consistency tokens tied to content versions, so checks are evaluated at a snapshot at least as new as the content.

Open-source descendants: SpiceDB, OpenFGA, Ory Keto. Airbnb Himeji (2022) is an in-house Zanzibar-inspired system.

17.3 🌍 In production: multi-tenancy

Salesforce: the classic metadata-driven multi-tenant design. Every row carries an org ID, tenants share huge database instances, and per-tenant customizations are stored as metadata, not schema changes.
Slack Enterprise Key Management (EKM): customers control the encryption keys for their messages and files (in their own AWS KMS), and can revoke access. It's a pricing-tier feature backed by architecture.
Shopify pods (§20.7) and AWS cells: tenant isolation at the infrastructure level.
Postgres row-level security (RLS): enforces tenant_id = current_setting('app.tenant') in the database, as a second line of defense against application bugs that forget a WHERE tenant_id = ?.

17.4 ⚠️ Corner cases & deep details

JWT revocation: a fired employee's 1-hour JWT keeps working for an hour. Use short TTLs, a revocation list checked at the gateway for high-risk actions, and refresh-token rotation with reuse detection.
IDOR (insecure direct object reference): GET /invoices/1234 is authenticated but not authorized, so any user can enumerate IDs. Every data access checks ownership. Non-guessable IDs are defense in depth, not the fix.
Confused deputy: service A calls B with A's broad credentials on behalf of a user, and B can't tell whether the user is allowed. Propagate the end user's identity (token exchange, on-behalf-of flows).
Background jobs without tenant context: a nightly job iterates over all data with superuser rights, and one bug mixes tenants. Run jobs per tenant, with scoped credentials.
Caches and search indexes without the tenant in the key: a cross-tenant data leak. Put the tenant ID in the key and in the index filter.
Erasure vs backups: GDPR deletion must also cover backups, logs, analytics, and ML features. Crypto-shredding (encrypt per-user or per-tenant data with its own key, then delete the key) makes deletion from immutable backups feasible.
Key rotation: rotating JWKS keys without overlap invalidates every live token. Publish the new key before you start signing with it, and keep the old key until all tokens signed with it have expired.