Chapter 1 sold the idea of a reliability budget; this chapter is how you actually build one — what to measure, where to set the target, and how to know you're burning budget too fast before the month is gone.
Last chapter ended with a promise: a single number both teams agree to, and a budget of allowed failure to spend. Lovely. Now build it for real.
Three things have to be true before that budget does any work. You need a number that actually tracks whether users are happy. You need a target on that number that isn't just "whatever we happened to hit last year." And you need to spot trouble fast enough to do something about it — not in the postmortem, but while the budget's still draining.
That's this chapter. We turn feelings into ratios, learn why the average is a liar, figure out where to set the line, write down what happens when you cross it, and finish on the most counterintuitive bug in the whole field: being too reliable.
"Is the service OK?" is a feeling. Someone squints at a dashboard, the graphs look green-ish, and they say "yeah, seems fine." That's not something you can put in a budget, page on, or argue with.
So you turn the feeling into a number, and the number is embarrassingly simple: good events divided by total events. What fraction of checkouts succeeded? What fraction of page loads came back in under 300ms? A value between 0 and 1 that goes up when users are happy and down when they're not. That ratio is called a Service Level Indicator — an SLI. The whole sophistication is in choosing which events count as "good."
And here's the rule that separates a useful SLI from a useless one: follow the user's journey, not the machine's internals. "Did checkout work, and was it fast?" is a user journey. CPU load at 80% is a machine internal — no user has ever filed a ticket because your CPU was busy. They file tickets because the page hung. Measure the hang.
You don't have to invent SLIs from scratch. Most services fall into one of three shapes, and each shape has a short, well-worn list of things worth measuring.
The temptation is to measure all of them, for everything. Resist it. Pick one to three SLIs per user journey. A dashboard with 50 SLIs isn't 50 times as informative — it's a wall of numbers nobody reads, where the one that matters is hiding in plain sight next to forty-nine that don't. Fewer, sharper indicators beat a comprehensive blur.
Here's the most expensive mistake in monitoring. You measure latency, you take the average, the average is 80ms, you sleep soundly. Meanwhile a slice of your users are waiting four seconds for the same page, and they're churning. The mean told you nothing, because the mean hides the tail — and the tail is exactly where users suffer.
The fix is percentiles, which sound scarier than they are. The p99 of latency is just "the experience of your unluckiest 1%" — sort every request by how slow it was, walk 99% of the way up, and read off the number. One request in a hundred was at least this slow.
And the cruel twist: your unluckiest 1% is rarely random. Your heaviest users make the most requests, and every request is another lottery ticket for the slow one. More tickets, more likely to hit the tail. So the people having the worst experience are often your most valuable ones. Alert and target on percentiles, never on means.
Two honest engineers can measure "the same" SLI and get different numbers, and neither is lying — they're standing in different places. Every vantage point shows you a different slice of reality, and crucially, each one is blind to something specific. Picking where to measure is picking which blind spot you can live with.
Server-side logs are easy — the data's already there — but they can only count requests that arrived. The user whose connection timed out before it reached you simply doesn't appear; from the server's view, that outage never happened. Move outward and you see more truth and accept more cost.
| Vantage point | What it sees | What it's blind to |
|---|---|---|
| Server / app logs | Requests that reached your code, cheaply and in detail. | Everything that never arrived — drops, timeouts, DNS failures. |
| Load balancer | Closer to the edge; counts requests the app dropped before logging. | The last mile — the user's wifi, their browser, their continent. |
| Client-side | The user's actual reality, slow networks and all. | Hard to collect, sampled, and noisy with problems you can't fix. |
| Synthetic probes | A consistent heartbeat that runs even at 3am when no users do. | Real users — a probe is a guess about what they'd experience. |
Now you have an SLI. Where do you draw the line on it — that's the SLO, the Service Level Objective. The instinct is to look at what you've been hitting — "we ran at 99.97% last quarter, let's promise 99.97%" — and that instinct is wrong. Your historic number is a fact, not a decision. Start instead from the question that actually matters: how much unreliability will users tolerate before they're unhappy? Set the line there.
Then give yourself room. Promise less than you can deliver, so a bad week doesn't instantly blow the SLO. It helps to keep two numbers straight: the aspirational SLO (where you'd love to be) and the achievable SLO (where you actually live). And the lawyer's version — the SLA, with money attached when you miss — must sit below both. Never let the contract promise more than the engineering can keep.
Chapter 1 sold you the budget. Here's the part everyone skips and then regrets: writing down what happens when you spend it — in advance, in calm daylight, before anything is on fire.
The error budget policy is a short document that answers the awkward questions before they're awkward. What happens at 50% burn? At 100%? Who has the authority to call a release freeze — and just as importantly, what conditions thaw it again? You decide all of this when nobody's adrenaline is up, because that's the only time you'll decide it honestly.
And it gets signed — by dev, by SRE, by product — so that when the freeze triggers, it's a deal everyone already agreed to, not a fight someone has to win. Callback to Chapter 1: a budget without a pre-agreed consequence is a suggestion box. The signature is what turns it into a contract.
Most alerting is broken in the same way: it fires on "errors are high." But high compared to what? A brief spike at 2% errors might be fine; a steady 0.5% might be quietly eating your whole month. The number that actually matters isn't the error rate — it's how fast you're spending the budget. That's the burn rate.
The intuition is clean. Burn rate 1× means you're spending the budget at exactly the speed that makes it last the whole window — right on plan. Burn rate 14× means you're spending fourteen times too fast: a 28-day budget gone in about two days. Now the question "should I wake someone?" has a numeric answer.
So you set up two alerts with different personalities. A fast-burn alert — short window, high multiple — pages someone, because at this rate the month evaporates in days. A slow-burn alert — long window, low multiple — files a ticket, because a slow leak is real but nobody needs to lose sleep tonight.
One refinement saves your sanity: require two windows to agree before paging — say, the last 1 hour and the last 5 minutes both over threshold. The long window proves it's significant; the short window proves it's still happening. A 5-minute blip that's already over won't pass the long window, so it never wakes anyone.
&&. Both windows must clear 14× — the long one to prove it's real, the short one to prove it's still happening, so the page stops once the incident does.Here's the twist nobody expects the first time. Google runs a lock service called Chubby that half of the company depends on. For a long stretch it simply never went down — it sailed past its SLO month after month. Sounds like a triumph. It was a trap.
Because when something never fails, everyone quietly starts assuming it can't. Teams built on Chubby as if it had no SLO at all — no fallback, no graceful degradation, no plan for the day it blinked. Chubby was making a promise of near-perfect reliability that nobody had actually agreed to, and the whole company had taken out a loan against it. Then, one day, it blinked. Cascading mess.
Google's fix is gloriously perverse: deliberate, planned outages to burn off the surplus budget and bring Chubby back down to its actual SLO. Controlled downtime keeps dependents honest — they keep their fallbacks working because they keep needing them. The lesson generalizes: if you're wildly outperforming your SLO, you're not a hero, you're over-delivering on a contract you never signed, and someone downstream is about to build on a promise you can't keep forever.