Most outages aren't caused by hardware or hackers — they're caused by us, shipping changes. You can't stop changing, so the move is to make change so safe and routine that nobody holds their breath on deploy day.
Here's an uncomfortable statistic the SRE book keeps coming back to: the leading cause of outages isn't a meteor through the datacenter roof. It's a change. A deploy. Someone pushed something.
Which is awkward, because shipping is the whole point — a service that never changes is a service nobody's improving. So the goal isn't to stop changing. It's to make each change small, reversible, and dull. This chapter is about turning "ship it" from a weekly adventure into a non-event you can do at 4pm on a Friday without a single Slack message starting with "hey so".
Programmer framing: think of your deploy pipeline as a function. You want it pure (same input, same output), composable (small steps you can reason about), and reversible (an undo that always works). Everything below is one of those three properties, dressed up in operations clothes.
When something breaks, the instinct is to look outward — a flaky disk, a DDoS, a cloud provider having a bad day. But pull the postmortems and the pattern is brutal: most outages trace back to a change we made. New binary. New config. A flag flipped. The villain is usually us, last Tuesday, at 4:55pm.
The naive fix writes itself: change less. Freeze deploys, add sign-offs, make Friday holy. And it works the way amputating your legs works as a fix for stubbed toes — the outages stop, and so does the product. The changes still pile up behind the dam; they just arrive later, bigger, and angrier.
The SRE move is the opposite of a freeze: change smaller and safer, more often. Make a release so routine and so reversible that one going wrong is a shrug, not a saga. At Google this isn't a vibe — it's a job title. Release engineering is a real discipline whose entire mission is making "ship it" a solved problem instead of a weekly roll of the dice.
A build is a function: source in, bits out. The problem is that most builds secretly take a second, invisible argument — whatever happens to be installed on the machine running them. The compiler version. A system library. An env var someone set in 2019. So the same source gives different bits on different laptops, and "works on my machine" becomes a real, load-bearing sentence.
A hermetic build closes that hole: same source plus same build equals the same bits, on any machine, today or three years from now. You do it by pinning everything — the compiler version, every dependency, down to the transitive ones — so the build depends on nothing it didn't bring with it. "Works on my machine" stops being an excuse and becomes what it always was: a build-system bug.
Why an SRE cares about a property that sounds like a build-team chore: reproducibility is what makes a rollback trustworthy. When you roll back to "the version from before lunch," you need that to be exactly the version from before lunch — bit for bit — not "a fresh build of the same commit that might pick up a different libssl." A rollback you can't trust isn't a rollback; it's a second deploy you're doing while on fire.
build(source) always returns the same artifact, with no hidden reads of ambient state. Once builds are pure, rollbacks become honest.Picture two releases. One bundles 3 changes; the other bundles 300. Both break in production. The first is a five-minute git log read — three suspects, the diff fits on a screen. The second is an archaeology dig: 300 changes, all landed together, and the bug is some interaction between two of them that nobody will find before sunrise.
This is the whole argument for shipping often. Frequent releases shrink the suspect list (fewer changes per deploy) and the blast radius (less new code hitting users at once). They also keep the deploy muscle warm — a team that ships twenty times a week has a boring, well-worn path; a team that ships once a quarter rediscovers every sharp edge each time, in the dark.
The reflex says speed and safety are a tradeoff — go fast and you'll break things. Backwards. The dangerous thing is the big batch. Small, frequent releases are both faster and safer, because each one is individually trivial to reason about and to undo. Slow and careful, done in giant lumps, is the actual risk.
The name is grim and instructive. Coal miners carried a canary into the mine; if the air went bad, the bird keeled over first, and the miners got out before they did. A canary in a deploy is the same trade: you give the new version to a tiny slice of real traffic, watch it like a hawk, and if it keels over, only the canary's worth of users got hurt.
What do you watch? The instruments you already built. The four golden signals from Chapter 1 — latency, traffic, errors, saturation — on the canary versus the rest of the fleet. And the error budget burn from Chapter 2: a canary that's torching budget faster than baseline is a canary that's dying.
Why bother, when you have a thorough test suite? Because real traffic contains things your tests never will. Users send malformed input, hit scale you didn't simulate, and trip dependency behavior that only shows up under real load. Your test data is polite; production traffic is feral. The canary is how you let the feral stuff find your bug while it can only cost you 1% of users instead of all of them.
The canary passed. Don't celebrate by slamming it to 100% — that throws away everything the canary just bought you. Instead, widen in stages: 1% → 10% → 50% → 100%, with a health gate between each step. The new version only advances if the signals at the current stage still look clean.
The point is that each gate is automated. A regression in the golden signals trips the gate, halts the rollout, and rolls back — no human courage required, no 3am judgment call about whether that error spike is "probably fine." The machine decides, and it decides the boring way: when in doubt, stop.
And the blast-radius math is the payoff. A bug caught at the 1% gate touched 1% of users for the couple of minutes before the gate tripped. The same bug shipped straight to 100% touched everyone, for as long as it took a human to notice, wake up, and react. Same bug, two wildly different incidents — and the only difference is the staircase.
Here's the single test that tells you whether your deploy system is any good. It's 3am. The newest person on the on-call rotation, six weeks into the job, sees the new version misbehaving. Can they roll it back, right now, with one command, without waking anyone up to ask permission? If yes, you have a deploy system. If no, you have a deploy ritual and a future war story.
The trap under pressure is "fix forward" — leaving the bad version live and trying to patch it in place while users suffer. It feels brave. It's how a ten-minute blip becomes a four-hour outage, because now you're debugging a live system, under stress, with everyone watching, and your "quick fix" is itself an untested change going straight to 100%.
The rule, which Chapter 6 will hammer harder: roll back first, debug at brunch. Restore service with the known-good version, get users out of the blast radius, then investigate calmly with coffee and no audience. Mitigate first; understand later. A boring rollback is the difference between an incident and an anecdote.
Notice that two things we keep saying together are actually separate. Deploy is "the new code is now running on the servers." Launch is "users can now see the new thing." A feature flag drives a wedge between them: the code ships dark — present but switched off — and a flag decides who, if anyone, gets to see it.
Now you can launch on a dial instead of a switch. Flip the feature on for 1% of users, watch, go to 10%, watch, then everyone — a progressive rollout for behavior, decoupled from the binary push. And the flag is the fastest rollback you own: something's wrong, you flip it off, done. No build, no redeploy, no canary — just a config value flipping from true to false and the new thing vanishing.
The tax, because everything has one: flags accumulate. Six months later you've got 200 of them, half permanently on, a quarter permanently off, and nobody remembers which. Dead flags are dead weight — branches in your code that exist for no reason. Prune them like you pruned the noisy alerts in Chapter 4: a flag that's been at 100% for a quarter isn't a flag anymore, it's just code pretending to be configurable.
rampPercent = 0 is your panic button — the fastest rollback in the building.There's a comforting lie that a config change is "safer" than a code change — it's just data, after all, just a little YAML. Tell that to the postmortems: some of Google's largest outages were a single config push gone wrong. Config takes down production exactly as well as binaries do. The blast radius doesn't check whether the bytes came from a .scala file or a .yaml one.
So configs get the same rules as code, no exceptions. Version it — every change in source control, with history and blame. Review it — a second set of eyes, same as a pull request. Canary it — roll it out in stages behind health gates, just like a binary. A config push is a deploy. Treat it like one.
If anything, a one-line YAML change deserves more paranoia than a thousand-line refactor — arguably a lot more. The refactor went through the compiler, the test suite, and a careful review because it looked scary. The YAML looked harmless, so nobody load-tested it, nobody type-checked it (YAML doesn't type-check, whatever you've heard), and it sails straight into production where it's parsed live for the first time. The scariest change is the one that didn't look like a change.