Site Reliability Engineering · ch.10 · launches & capacity
📟 Chapter 10 · Surviving your own success

A launch is a planned emergency

The traffic spike you spent six months begging for arrives on a schedule you chose — and it will find every weakness the previous nine chapters warned you about. This is the finale: capacity, checklists, and shipping the thing without becoming your own outage.

This is the last stop. Nine chapters of measuring honestly, automating the grind, and surviving overload — and now we point all of it at the one day that's guaranteed to test every assumption at once: launch day.

A launch is the moment your service meets the traffic you've been forecasting in spreadsheets. It's also the moment you discover whether those forecasts were fiction. The good news for an engineer: a launch is one of the few emergencies you get to schedule. The bad news: people treat scheduled emergencies like celebrations and forget to bring the fire extinguisher.

So this chapter is about turning the chaos into a checklist — provisioning for the worst Tuesday, testing the wall before users find it, and ramping visibility so a bad release embarrasses 1% of users instead of all of them. And then we walk the whole arc back: the entire pager, one idea at a time.

1The best day and the worst day

Launch day is both at once. Maximum attention — everyone's watching, the blog post is live, the press release went out. Maximum traffic — the most load this code has ever taken. And minimum operational history — you have zero days of production experience with exactly this combination of code, config, and crowd.

Here's the part nobody believes until it happens to them: success kills more launches than bugs do. The app that went viral on Friday and fell over on Friday night didn't fail. It succeeded — harder and faster than its capacity plan ever imagined. The code was fine. The forecast was a polite fiction, and reality declined to be polite.

The SRE reframe is the whole chapter in one sentence: a launch is a traffic emergency you scheduled. You wouldn't walk into a known outage without a runbook, a rollback, and someone on the pager. A launch is exactly that, minus the surprise — so prepare for it like the emergency it is, while you still have the luxury of time.

📟
The launch that dies of success is the cruelest postmortem to write: every line is "the thing we wanted, but more." Treat the demand you're hoping for as a fault you're planning around.

2The launch review — a checklist with a pulse

Google's answer to launch-day chaos isn't a heroic firefighter. It's a person with a checklist. They have a grand name — Launch Coordination Engineering — but the job is humble: a consulting SRE who audits your launch before it ships and asks a list of deeply boring questions.

The checklist's greatest hits, and you'll notice every one points back at a chapter you've already read:

  • Load tested at expected peak ×3? (section 5)
  • Quota secured from every dependency you call? (your launch is their traffic spike too)
  • Rollback plan tested, not just written? (Ch.5)
  • Feature-flagged so you can turn it off without a deploy? (Ch.5)
  • Degraded mode defined for when a dependency dies? (Ch.8)
  • Alerts and dashboards live before launch, not after? (Ch.4)
  • Someone on the pager who actually knows this system? (Ch.6)

None of these are clever. That's the point. Every checkbox is a past outage wearing a disguise — somebody, somewhere, shipped without it, and the box exists because of how that went.

Interactive · launch day roulette Pick 3 prep items · then launch · survive what comes

Budget for 3 of 6. (Of course you only get three. There's always a deadline.)

Pick three prep items, then hit Launch.
0 / 0 outage-free

3Capacity planning — the slope and the steps

Demand comes in two flavors, and they need completely different treatment. Organic growth is the steady slope: users trickling in week over week, the line you can fit to history and extrapolate. It's boring and it's forecastable, which makes it the easy half.

Inorganic spikes are the hard half: launches, marketing pushes, a celebrity tweet, a feature that lands on the front page. These are step functions, and the cruel thing about a step function is that you cannot forecast it from history — there's no slope to extend. You have to be told. That's the whole reason launch calendars exist and capacity teams sit close to product: the only defense against a surprise step is to make it stop being a surprise.

So the recipe: forecast the slope, reserve headroom for the steps you've been warned about, and respect lead time. Capacity you order today doesn't arrive today — machines take weeks, sometimes a quarter or more. Your capacity plan lives a few quarters out by necessity, which means the demand you're provisioning for is demand you can only guess at. Welcome to the job.

4N+2 — provisioning with a built-in bad day

Here's the provisioning rule, and it has a bad day baked right into it. Run enough capacity that you can lose one site to planned maintenance AND one site to a surprise failure — at the same time — and still serve peak traffic. That's N+2: N sites to carry the load, plus two you can afford to lose.

The tempting cheaper option is N+1, and it sounds perfectly safe until you remember that maintenance is not optional. You will take a site down to patch it — that's not a failure, it's a Tuesday. The moment you do, your one spare is spoken for, and N+1 quietly becomes plain N with extra paperwork. One hiccup during that window and you're serving an outage.

So the plain-English version, no acronym required: you must serve peak with two sites dark, always. One dark for the work you planned, one dark for the surprise that arrives mid-maintenance because the universe has a sense of timing.

Interactive · the N+2 planner Set the fleet · darken sites · read the verdict
sites 5
capacity / site 40 u
peak traffic 120 u
surviving capacity
peak demand
verdict
💡
Two functions, two jobs. survivable subtracts two sites before it does the math — the bad-Tuesday clause in code. monthsOfHeadroom turns your growth slope into a countdown, because capacity ordered today arrives in months, and that number is your alarm clock.

5Load testing — find the wall before your users do

A load test is synthetic traffic you fire at your own system to find the wall first — on a quiet afternoon, with no users watching, while it's still cheap to learn the answer. The alternative is finding the wall at peak with real customers, which is also called "an outage."

The catch: you have to test the real system, not a lab toy. Real dependency calls, not mocks that always say yes. Real data shapes, not three tidy rows. And your degraded modes actually exercised — yes, the Ch.8 untested degraded path again, because a fallback you've never run under load is just a comment that lies. The whole value of the test is exercising the parts you're nervous about.

What you're hunting for is the knee of your own hockey stick (Ch.8): the point where one more request per second tips latency off a cliff. Find it while it's cheap. The number you walk away with — requests per second at acceptable latency — isn't a trophy. It's the input every other section here depends on: it's the N in N+2, it's the capacity line every ramp is measured against.

6Dark launches and ramps — the gentlest way in

The gentlest possible launch is one your users can't even see. Send real production traffic through the new code path — but throw the results away instead of showing them. That's a dark launch: you get real load, real weird inputs, real dependency behavior, and real bugs surfacing — at zero user-visible risk, because nobody's looking at the output. It's a load test made of genuine traffic, which is the only kind that doesn't lie.

Once you trust the code, you ramp visibility the Ch.5 way: 1% → 10% → 50% → 100%, all behind a flag, each step a chance to abort before the blast radius grows. A bad release at 1% is a footnote. A bad release at 100% is a status page.

And the marketing department has been doing this forever — they just dress it differently. A waitlist, gradual invites, "rolling out over the coming weeks" — that's a ramp in a nicer outfit. A queue, after all, is just a load shedder with an aesthetic (Ch.8, hello again): it caps intake to what you can actually serve, and calls it exclusivity instead of backpressure.

Interactive · the ramp room Pick a launch strategy · watch demand meet capacity
hours over capacity
users turned away
users served badly

7The whole pager — a finale recap

Ten chapters, one bet: reliability is engineering, not heroics. Walk the arc back, one idea apiece:

  • Ch.1–2 · budgets instead of arguments. A shared number ends the dev-vs-ops war that no meeting ever could.
  • Ch.3 · automation instead of toil. If a human does it twice a week, a human already failed to write the script.
  • Ch.4 · symptoms instead of noise. Page on what users feel, not on every twitchy graph.
  • Ch.5 · boring deploys. Change causes most outages, so make change small, reversible, and dull.
  • Ch.6 · humans designed-for at 3am. The pager is a system too — staff it so it doesn't burn people down.
  • Ch.7 · blameless lessons, collected. Blame the system, write it down, never waste an outage.
  • Ch.8 · overload survived on purpose. Shed load, degrade gracefully, push back before you fall over.
  • Ch.9 · restores, not backups. A backup you've never restored is a rumor.
  • Ch.10 · launches that emergency-drill themselves. Schedule the spike, rehearse the bad Tuesday, ramp the blast radius.

And the callback the whole course was built around: hope was never the strategy. Budgets, drills and checklists were.

📟
You started at "100% is the wrong target" and ended at "serve peak with two sites dark." Same idea wearing different clothes: name the failure you'll accept, then engineer around it on purpose. That's the entire field.

8Check yourself

3 questions · instant feedback 0 / 3