They want restores — the backup is just the receipt, and an untested receipt is a prayer. This chapter is about the one failure you can't roll back: data that's simply gone.
Every other chapter in this book is about failures that end. The server comes back. The deploy rolls back. The pager goes quiet. You burned some error budget, you wrote a postmortem, life goes on.
This chapter is about the one that doesn't. When data is gone — actually gone, not just unreachable — there's no rollback, no retry, no graceful degradation. There's just the awful moment where you realize the thing you needed was overwritten three weeks ago and the backup that should have had it… didn't.
The good news is that data loss is one of the most preventable catastrophes in the business — if you stop thinking about it as a backup problem and start thinking about it as a restore problem. That one reframe is the whole chapter.
An hour of downtime is forgiven by lunchtime. Your users grumble, refresh, go get coffee, and by the time the status page goes green they've mostly forgotten it happened. Downtime is a verb that finishes.
Lost data is a noun that lasts forever. The wedding photos, the year of accounting, the only copy of the document — those don't come back when the incident ends, because for this kind of failure the incident never really ends. Users feel this difference in their gut, and they're right to. You can apologize your way out of an outage. You cannot apologize your way back to data that no longer exists.
That's why availability gets an error budget (Ch.1–2) — a generous, spendable allowance of failure — and durability effectively doesn't. The durability target isn't "four nines." It's "all the nines you can buy, and then a layer of paranoia on top of that." There is no budget for "oops, we permanently destroyed 0.01% of customer data this quarter."
Here's the trap that gets nearly everyone, including people who should know better. You've got your data on three replicas across two regions with synchronous replication. Surely that's backed up six ways from Sunday?
No. Replication is a very fast, very faithful copier. Its entire job is to make every copy identical to the primary as quickly as possible. So when you run DROP TABLE users at 14:02, replication does exactly what it's built to do: it copies that DROP TABLE to every replica at roughly the speed of light. Twelve milliseconds later, all three copies are equally, perfectly empty.
RAID, multi-region, consensus quorums — every one of them protects against hardware loss, and not one of them protects against your own mistakes. They all preserve garbage with the same diligence they preserve gold, because they can't tell the difference. To them a malicious write and a legitimate one are the same bytes.
A backup's defining feature is that it's behind — disconnected in time from your latest mistake. Yesterday's snapshot doesn't know about today's DROP TABLE, and that ignorance is exactly what saves you. The replication lag you're forever tempted to optimize to zero? In a backup, the lag is the feature.
Data dies in three distinct ways, and each one defeats a different defense. That's the whole reason a single defense is never enough — you're fighting three different enemies.
Notice the cruel asymmetry: the loudest disaster (infra failure) is the easiest to survive, and the quietest one (a slow bug) is the deadliest. Severity isn't what kills you. Detection time is.
Start with the most common flavor: somebody deleted something they didn't mean to. The fix is almost insultingly simple — make deletion reversible. A trash can with a clock: flag the record as deleted now, actually purge it in 30 to 60 days.
This is cheap, the restores are instant, and it handles the single most frequent case entirely. The user who deletes the wrong folder at 2pm gets it back at 2:01 with a click, and your real backup system never even hears about it.
If you write Scala, your instincts already point the right way. Deletion isn't an event that destroys data; it's a state the data is in. You're not removing the row — you're moving it into the None branch of an Option[T], with a timer on when the garbage collector finally gets to keep it. Reversible by construction.
DELETE. It should be an UPDATE … SET deleted_at = now(). Hard deletes are a job for a slow, boring, well-audited background reaper — never the live request path.Here's the line worth tattooing on the ops room wall: nobody was ever saved by a backup. They were saved by a restore. The backup is just the receipt. The restore is the thing that actually gets your data back, and the two are not the same — a backup can exist and be completely useless.
A backup you have never restored from is Schrödinger's data: simultaneously fine and worthless, and you won't know which until you open the box during your worst day of the year. The cron job that died in March, the format that changed, the encryption key nobody can find, the tape that's bit-rotted — all of these are invisible until the one moment you need the data and discover you don't have it.
So you treat restores as a first-class engineering process, not an afterthought: scheduled restore drills, automated verification that the restored data is actually correct, and an honest measurement of restore time. A restore that technically works but takes seven days is a second outage wearing a safety vest. Your customers won't wait a week, and "the data came back eventually" is not the win you think it is.
Recall Chapter 1: hope is not a strategy. A backup you've never restored from is hope in a tarball. The only backup that counts is the one you've already proven you can bring back.
Soft deletion handles the oops. Backups handle the disk fire. Neither one helps against the killer from section 3 — the slow, silent corruption bug — because both of those defenses assume you know something is wrong. Against silent corruption, you don't. The clock is running and nobody has noticed.
So you add a third layer whose only job is to notice faster. Out-of-band validators run continuously, off to the side: they checksum your data and diff it between systems — primary against replica, database against its index, today's row count against yesterday's. When the numbers disagree, they shout. That shrinks the discovery window from weeks down to hours.
And shrinking discovery is worth more than it looks, for two compounding reasons. The earlier you catch corruption, the less history you have to rewind to undo it — and, crucially, the more of the damage is still inside your backup retention, which means still recoverable. Catch it in hours and yesterday's backup is clean. Catch it in months and the clean backup expired weeks ago.
RetentionPolicy.safe is the whole chapter in one boolean — your backups have to remember for longer than your bugs take to discover. If that returns false, you don't have a backup system; you have a countdown.Time to make the killer concrete. A bug starts silently corrupting data on day 0. The question that decides whether you ever recover is brutally simple: on what day does someone finally notice?
Drag the discovery day. Toggle your defenses. Watch the rust spread — and watch how much of it you can still claw back versus how much is gone for good.
Here's the single rule that ties the whole chapter together, and it fits in one sentence: you can only recover what your retention still remembers, so retention must outlast discovery.
Run the numbers honestly. If your application bugs typically take 60 days to notice, and you keep 30 days of backups, then you do not have a backup system. You have a 30-day grace period before permanent loss — a countdown that started the moment the bug shipped, and that you'll usually lose. By the time you discover the corruption, the last clean backup expired a month ago.
The instinct is to fix this by buying more retention. You can — storage is cheap-ish, and 90 days beats 30. But the better lever is the one from section 6: shorten discovery. A validator that drops discovery from 60 days to a few hours makes a 30-day retention luxuriously safe, for far less money than hoarding nine months of cold storage. Design your retention around discovery time, not around what storage costs.