24. Backup & Restore as a Discipline

Nobody needs backups. Everybody needs restores. A backup that has never been restored is a hope, not a control.

Backups show up in interviews as "what's your RPO/RTO?", as a follow-up on any stateful design ("what if someone runs DELETE without a WHERE?"), and in Staff+ and engineering-manager questions about ransomware and resilience. This section treats backup and restore as an engineering system with SLOs, architecture, testing, and failure modes.

24.1 Replication is not backup

24.2 Define restore objectives per data class

Two things candidates forget: derived data shouldn't be backed up at all (rebuild it, but measure rebuild time, which is your real RTO), and platform state (identity provider, DNS, secrets, IaC, CI/CD) must be restorable first, or nothing else can be restored.

24.3 The 3-2-1-1-0 rule

 3  copies of the data (production + 2 backups)
 2  different storage media / services
 1  copy off-site (different region)
 1  copy IMMUTABLE or offline/air-gapped (object lock / WORM, separate account & credentials)
 0  errors in automated restore verification

24.4 Backup types and how they combine

The standard production setup for a relational database: a daily physical base backup, continuous WAL/binlog archiving (PITR window of e.g. 35 days), copies to another region and another account with object lock, a weekly logical export of critical tables for per-tenant restores, and an automated restore test every day.

24.5 Architecture: a backup & restore platform

 Production accounts                         Backup vault account (separate org unit,
 ─────────────────────                       separate credentials, break-glass access only)
 DBs ──base backups + WAL──▶ staging bucket ──copy──▶ Vault bucket (versioned, OBJECT LOCK,
 Object stores ─replication (versioned)──────────────▶   cross-region, KMS key owned by vault acct)
 K8s/IaC/configs/secrets ─exports──────────────────▶
 SaaS (email, docs, CRM) ─API exports──────────────▶
                                                        │
  Backup catalog (what, when, where, encryption key id, schema version, checksums) ◀──┘
                                                        │
 Restore verification service (daily, automated):
   pick random backup ─▶ restore into isolated env ─▶ checks: checksums, row counts,
   app-level smoke tests, PITR to random timestamp ─▶ record actual restore time vs RTO
   ─▶ alert on failure or RTO breach
 Isolated recovery environment ("clean room") for ransomware scenarios

Design principles:

• Separate the blast radius of credentials. Attackers and buggy automation delete backups first. Production admins must not be able to delete or shorten retention in the vault. Use a different account, MFA, and break-glass procedures.
• Encryption keys are part of the backup. If the KMS key is deleted or inaccessible, the backup is gone. Keys for backups live with the vault, with their own deletion protection (§56, §55).
• A catalog: you can't restore what you can't find. Record what each backup contains, its schema/app version, its key ID, and its checksum.

24.6 Restore engineering: the parts nobody practices

Restore time math (the RTO reality check):

 Database: 100 TB; restore throughput from object storage ≈ 1 GB/s (parallel streams)
   → 100 TB / 1 GB/s ≈ 100,000 s ≈ 28 hours  (+ WAL replay)
 Business RTO: 4 hours  → mismatch of 7×
 Options: shard so each shard restores in parallel (100 shards × 1 TB → ~17 min each);
          keep a delayed replica (e.g. 1 h behind) for fast "undo";
          storage snapshots for fast whole-volume restore + PITR for precision;
          tiered RTO: restore the most critical tenants first.

Restore granularity matters as much as speed. A single-tenant corruption in a shared database needs a per-tenant restore: restore to a side environment, extract that tenant's rows, and merge them back without overwriting other tenants' newer data. That has to be built and tested before it's needed.

Restore order (dependency graph):

 1. Identity provider / break-glass accounts   4. Databases (tier 0 → 1)
 2. Networking, DNS, certificates/PKI           5. Object data, queues
 3. Secrets / KMS, CI/CD, IaC state             6. Derived data rebuilds (indexes, caches)

24.7 🌍 Real incidents that define this discipline

24.8 Interview answer template

🧑‍💼 "Someone runs a bad migration that corrupts the orders table at 14:02. We notice at 15:30. What happens?"

🎤 "We don't fail over, because the corruption is already replicated. First, stop the bleeding: disable the writer or flag, and freeze schema changes. Then PITR the orders database into a side environment at 14:01, using the base backup plus WAL replay. Then diff the two: rows changed between 14:02 and 15:30 by the bad migration are repaired from the restore, while legitimate writes made after 14:02 (new orders) are kept. That's why we log changes with a request or migration ID. Re-run the downstream projections (search, caches, analytics) for affected rows. Our tested RTO for this restore path is about 40 minutes per shard, and we rehearse it quarterly. Afterwards: the migration process gets a dry run on a production clone, and a delayed replica (1 h behind) for faster undo."

24.9 ⚠️ Corner cases & deep details

• Restored data resurrects deleted personal data (GDPR): restoring a 30-day-old backup brings back users who exercised erasure since. Keep a deletion log and re-apply it after every restore, or use crypto-shredding so old backups are unreadable for erased users.
• Schema/application version skew: a 6-month-old backup doesn't match today's code. Store the schema version in the catalog, and keep migration tooling able to bring old backups forward.
• Key rotation and deletion: backups encrypted with a retired key need that key version to stay available for the full retention period. A scheduled key deletion silently destroys backups.
• Crash-consistent snapshots of multi-volume databases: snapshotting volumes one by one yields an inconsistent set. Use the DB's backup API, quiesce, or take multi-volume consistent snapshots.
• Sharded systems: per-shard backups at different times aren't a consistent global point (§9.8). Decide whether you need coordinated snapshots or per-shard PITR with application-level reconciliation.
• Corruption older than retention: silent corruption detected after 60 days with 35 days of retention can't be fixed. Use checksums on write, periodic scrubbing, and long-retention monthly backups.
• Backups of SaaS data: providers (email, documents, CRM) typically operate on a shared responsibility model, protecting their infrastructure, not your accidental deletions. Back up critical SaaS data via APIs.
• Backup load on production: backups during peak hours steal I/O. Take them from replicas, and throttle.
• Legal hold vs retention expiry: data under legal hold must not be deleted by lifecycle rules, so holds must override lifecycle policies.
• Restore permissions are a security risk too: restoring production data into a less-protected test environment leaks PII. Restore into equally protected environments, or mask the data.