45. File Sync & Storage (Dropbox / OneDrive / Drive)

Business context. Users trust you with their files. The outcomes are durability (never lose a file), seamless sync (the product is "it just works"), and storage cost per user, which decides whether a free tier is viable.

Requirements. Upload, download, and sync across devices; offline edits; conflict handling; version history; sharing; 500M users; large files (GBs); efficient sync of small changes.

Design.

Client (watcher + local index) ─▶ chunk file (content-defined chunking, ~4 MB) ─▶ hash each chunk (SHA-256)
   ─▶ "which chunks do you already have?" ─▶ upload only missing chunks (presigned URLs, resumable)
       → block storage (object store, content-addressed by hash → global dedupe)
   ─▶ commit new file version: metadata service (file → ordered list of chunk hashes, version, parent)
        - metadata DB: strongly consistent (sharded relational / Spanner-class), per-user/namespace
   ─▶ notification service (long-poll/WebSocket) tells other devices "namespace changed at journal id N"
   ─▶ other devices pull the metadata delta since their cursor, then download missing chunks

Deep dives.

Content-defined chunking (rolling hash, e.g. Rabin): inserting bytes at the start of a file changes only the nearby chunks, not every chunk afterwards. Delta sync and dedupe depend on it.
Metadata is the hard part. Each user's or namespace's journal is an ordered log of changes. Devices sync with a cursor (the same idea as chat sync in §34).
Conflicts: two offline edits of the same version produce a "conflicted copy" file (simple and safe). Never silently merge binary files.
Durability: erasure coding (e.g. 6+3 Reed-Solomon) across AZs gives high durability at ~1.5× storage overhead, versus 3× for replication. Run background scrubbing that verifies checksums.
Cold data: most files are rarely read after 30 days. Tier them to cold storage, which is the biggest cost lever.

Strategic lens. Dedupe plus tiering plus erasure coding decides gross margin on storage. Enterprise features (OneDrive/SharePoint): retention policies, legal hold, DLP, ransomware recovery (restore a whole account to a point in time, which version history enables).

🏗️ Flow: saving a 2 GB file where only 1 MB changed.

 Watcher sees change (wait for file to be stable: no writes for N seconds / closed handle)
   ─▶ content-defined chunking (Rabin/FastCDC, ~4 MB average) → 512 chunks, hashes h1..h512
   ─▶ compare with last known version: only h37, h38 differ (insert shifted just local boundaries)
   ─▶ POST /commit_check {hashes} → server: "missing: h37', h38'"
   ─▶ upload 2 chunks (8 MB) to block store (content-addressed, encrypted at rest)
   ─▶ POST /commit {path, parent_rev=41, chunk list} → metadata svc: conditional on parent_rev
        ├─ ok → rev 42, journal entry (namespace N, journal_id 9001)
        └─ conflict (someone else committed rev 42) → save as "file (conflicted copy).ext"
   ─▶ notifier: namespace N changed → other devices pull journal since their cursor → fetch chunks

🌍 How it's built in production.

Dropbox Magic Pocket: in-house, exabyte-scale block storage with erasure coding, built after moving off S3 (§28.5).
Dropbox Nucleus ("Rewriting the heart of our sync engine", 2020): the desktop sync engine was rewritten in Rust around a clear data model (remote tree, local tree, synced tree) and heavily tested with randomized testing. The hard part of sync is correctness of the client state machine, not server throughput.
Azure Storage (SOSP 2011 paper) and Local Reconstruction Codes (USENIX ATC 2012): Microsoft's erasure coding reduces the number of fragments needed to rebuild a lost one, which cuts repair I/O. It's the storage substrate behind OneDrive-class services.
Meta Tectonic (FAST 2021): a single exabyte-scale multi-tenant file system that replaced several purpose-built storage systems, which shows the economics of consolidating storage platforms.

⚠️ Corner cases.

Case-insensitive file systems: Report.pdf and report.pdf are separate on Linux and the same file on macOS and Windows defaults. Detect and rename on conflict.
Illegal names and path lengths: CON, trailing dots, :, and paths over 260 characters on Windows. Sanitize per platform, and keep the original name in metadata.
Rename vs edit concurrency: device A renames a folder while device B edits a file inside it. The metadata model must use stable IDs (file IDs separate from paths) so both changes apply.
Moving a folder with 1M files: that must be one metadata operation (re-parent a node), not a million path updates.
Files being written while syncing (databases, VM images): wait for quiescence or skip known volatile types. Never sync a half-written file as the new version.
Ransomware mass encryption: thousands of files change to high-entropy content. Detect the pattern and offer restore the account to a point in time, which version history plus immutable chunks make possible.