Sync files,
not whole files

Start with the plumbing. A client (your laptop, your phone) talks to an API gateway — the single front door that authenticates you, checks quotas, and routes your request on. Behind it sit two very different jobs: storing the actual file bytes, and storing the small facts about the file (its name, owner, size, the list of blocks it's made of). The bytes go to block servers fronting cheap cloud storage; the facts go to a metadata database.

Uploading a small file is a one-shot affair: send the bytes, the block servers stash them, the metadata DB records the new file. Easy. The problem is a 2 GB video over hotel Wi-Fi — if the connection drops at 90%, you do not want to start over. So large uploads use a resumable upload: the client first asks the server "I want to upload this file, give me a session," then sends the file in chunks, each acknowledged. If the connection dies, the client asks "how far did you get?" and continues from there.

Download is the mirror image. The client asks the metadata DB "what is this file made of?" and gets back an ordered block list — a recipe of block hashes. It then fetches each block (often from a CDN or block-server cache, in parallel) and reassembles the file locally. Notice the split already paying off: a tiny metadata lookup tells you exactly which bytes you need, and you only fetch those.

Here's the core move. Don't store a file as one big blob. Chop it into fixed-size pieces — say 4 MB each — and store every piece independently. A 50 MB file becomes 13 blocks. The file itself is now just an ordered list of references to those blocks, plus a name and some metadata. These pieces are blocks (Dropbox calls the technique block-level storage).

Why bother? Two big payoffs fall out immediately:

• Partial transfer. Because each block stands alone, you can upload, download, and verify them one at a time and in parallel. Lose the connection mid-file and you've only lost the in-flight block, not the whole thing.
• It sets up everything else. Once a file is a list of blocks, the next three sections — delta sync, deduplication, versioning — are all just operations on that list. They'd be impossible (or wildly inefficient) on one opaque blob.

The last ingredient is how we name each block. Instead of a random ID, we name a block by the hash of its contents — run the bytes through SHA-256 and use the result as the block's address. Identical bytes always produce the same hash, so the name is the content. This is called content-addressable storage, and it's the quiet hero of the whole design: it makes "did this block change?" and "have we seen this block before?" the exact same question.

Now the payoff from the intro. When you save an edited file, the client re-chops it into blocks and re-hashes each one. Then it lines up the new block hashes against the old ones it already synced and asks a simple question per position: did this block's hash change? The blocks whose hashes differ are the only ones that need to travel; the rest are already in the cloud, byte-for-byte identical. Plain English: chop the file into fixed-size blocks, and when you edit, only re-upload the blocks that changed. The fancy name is block-level delta sync.

The server, meanwhile, never needs the whole file. It receives the handful of changed blocks plus the new ordered block list, stores the new blocks, and records "version N of this file is this list of hashes." To reconstruct any version, you just walk its block list and fetch each block — most of which it already had.

The bandwidth math is dramatic. Edit one paragraph in a 50 MB document and a whole-file upload moves 50 MB; delta sync moves one 4 MB block (or far less, with smaller blocks). For a file people touch ten times a day, that's the difference between a sync tool that's pleasant to use and one that pegs your uplink. Try it below.

Content-addressing hands us a second gift for free. If a block's name is the hash of its bytes, then two identical blocks — anywhere, from any user, in any file — have the exact same name. So when the server is told to store a block, it first checks: do I already have something at this hash? If yes, it doesn't store a second copy; it just points the new file's block list at the block that's already there. This is deduplication.

Think about how much this saves. A company circulates the same 30 MB onboarding PDF to 1,000 employees, all of whom save it to Drive. Naively that's 30 GB. With dedup it's 30 MB stored once, plus a tiny per-user reference. The same applies to the unchanged blocks of a file's history, to popular installers, to that meme everyone forwards.

But shared storage raises a question: if everybody points at one stored block, when is it safe to delete? You keep a reference count — how many files currently point at this block. Adding a file that uses the block increments it; deleting a file decrements it. Only when the count hits zero is the block truly orphaned and eligible for garbage collection. Watch the counts move below.

The block storage holds the bytes; the metadata database holds the story — and that story is what users actually interact with. It's a handful of related tables:

• files / namespaces — a file's name, owner, parent folder, size, current version pointer.
• versions — for each file, a chain of versions, each one stamped with a version number and a timestamp.
• block mappings — for each version, the ordered list of block hashes that reconstruct it (the "recipe").
• sharing / ACLs — the user→file relationships: who owns it, who it's shared with, what they can do.

This data is small per file but there's an enormous amount of it, so it gets sharded (Chapter 1 again) — usually by user or by namespace, so all of one person's files land together and a lookup never has to fan out across the whole fleet. And unlike the blocks, metadata demands strong consistency: when you rename a file or move it to the trash, every device must immediately agree on the new truth. A stale block is harmless (it's content-addressed, it can't be wrong); a stale file list shows you a file that no longer exists. So metadata gets the careful, consistent treatment; bulk bytes get the cheap, relaxed one.

Here's the genuinely hard part. You edit notes.txt on your laptop while it's offline on a plane. Meanwhile your phone, also out of sync, edits the same file. Both come back online and try to push their version. They started from the same base version but disagree about what comes next. That's a conflict, and how you handle it is the difference between a tool people trust and one that silently eats their work.

The mechanism for detecting it is a number. Every file version carries a version number (or, when changes can come from many independent devices, a vector clock — a per-device counter we met back in Chapter 7). When a client pushes, it says "I'm updating version 5." If the server is already on version 6, the client's edit was based on stale ground — the version numbers don't line up, so the server knows a conflict happened rather than blindly overwriting.

What you do next is a policy choice. First-write-wins accepts whoever arrived first and rejects the rest. Silent last-write-wins is tempting but dangerous — it throws away someone's work without telling them. The safe, user-friendly default is versioning with conflict copies: keep both. The server saves one as the new version and writes the other as a clearly-labeled notes (conflicted copy from phone).txt, then lets a human decide. Nobody's bytes are lost. Step through it below.

One device just uploaded a new version. How do your other devices find out — quickly — without hammering the server every two seconds asking "anything new? anything new?" This is the change notification problem, and it's a callback to the real-time messaging ideas from Chapter 10.

The crude approach is polling: each client asks "any changes since version 5?" on a timer. It works, but it's wasteful and laggy — too frequent and you melt the server, too rare and sync feels sluggish. Long-polling improves it: the client asks and the server holds the request open until there's actually something to report (or a timeout), then answers and the client immediately asks again. For something more instant, a WebSocket keeps a persistent two-way connection open so the server can push "files changed!" the moment it happens.

In practice a notification service sits between the upload path and the clients: when a version is committed, it fans out a small "your file X changed, go pull version N" message to every device subscribed to that file. The message is just a nudge — the device then runs the normal metadata-lookup-plus-block-fetch download. And for devices that are offline when the change lands, the notification goes into an offline backup queue (Chapter 1's message queue, again) and gets delivered when they reconnect, so nobody misses an update.

Step back and the two-store split keeps paying off. Metadata gets strong consistency via primary-replica replication: writes go to a primary that holds the authoritative version number, reads spread to replicas, and a failover promotes a replica if the primary dies. Because the version number is the single arbiter of "what's current," every device converges on one truth even as edits fly in from everywhere.

Versioning isn't just for conflicts — it's a feature. Since every version is a cheap block list over mostly-shared blocks, keeping a file's full version history costs almost nothing (you only ever stored the changed blocks). "Restore previous version" becomes "point the file at an older block list." For blocks that are old and rarely touched, you tier them down to cold storage — slower, far cheaper archival storage — and only thaw them on the rare restore.

And it all has to survive failures. Blocks get replicated across machines and regions so losing a disk loses nothing. The metadata DB fails over to a replica. The notification service degrades gracefully to polling. Here's the whole design as one checklist — the path a single saved edit takes:

1. 1Chop & hash — the client splits the file into fixed-size blocks and hashes each by content.
2. 2Diff — compare new block hashes to the last-synced ones; keep only the positions that changed.
3. 3Upload the delta — send just the changed blocks (resumably, if large) through the API gateway to the block servers.
4. 4Dedup — block storage keeps each content hash once, bumping reference counts instead of duplicating.
5. 5Commit metadata — record the new version (number + ordered block list) in the strongly-consistent metadata DB.
6. 6Resolve conflicts — if version numbers don't line up, keep both via a conflict copy; never silently overwrite.
7. 7Notify — fan out a small "version N exists" nudge over long-poll / WebSocket; queue it for offline devices.
8. 8Others pull — each device does a metadata lookup, fetches only the blocks it's missing, and reassembles.

Section 3 claimed that with content hashes, "what changed?" is just a comparison. Here's exactly that. Given the old list of block hashes and the new one, return the indices whose hash differs — those are the blocks to upload. No file contents, no bytes, no I/O: just lining up two lists of names. That purity is the whole point — it's why a client and server who never exchange a single byte can still agree on precisely what's missing. And the same hashes power dedup: store by hash, and identical blocks collapse for free. Flip between the languages; the shape is identical.