Site Reliability Engineering · ch.6 · on-call
📟 Chapter 6 · Holding the pager without burning out

3am you is a different person

The engineer who answers the page at 3am is slower, dumber, and more confident than the one who wrote the code at 3pm — and the system has to be designed for that person. This chapter is the math and psychology of carrying a pager sustainably.

Every reliable service eventually wakes someone up. On-call is the human fallback for the moments your automation didn't cover — and the person it wakes is operating on three hours of sleep and a spike of adrenaline.

That's the uncomfortable truth this chapter is built around. The 3am responder isn't the calm, careful engineer who wrote the code; it's a half-asleep version reaching for whatever pattern feels right. So the whole apparatus — the rotation math, the playbooks, the "roll back first, ask why later" reflex — exists to make that person's job survivable, and the system safe in their hands.

And there's the SRE twist you already met in Chapter 1: the people on the pager are engineers. They don't just suffer the page — they're expected to automate the next one out of existence. On-call is where toil announces itself loudest.

1The deal

Strip away the romance and on-call is one promise: somebody is guarding the SLO, and they'll respond within a set time. The primary answers the page in minutes — five, fifteen, whatever the service's SLO can stomach. The secondary is the backstop: they catch the page the primary slept through, and they're a second brain when one isn't enough.

The pager outranks your dinner, your movie, and your evening plans. It does not outrank your health or your sleep past the shift. That ordering is the whole ethic of a humane rotation: serious enough that you carry it everywhere, bounded enough that it doesn't carry you.

Here's the part that makes it SRE and not just operations. The person holding the pager is the same person who can change the code. So every incident is a complaint filed against the system — and the on-caller is expected to act on it: write the playbook, file the bug, build the automation that means nobody gets paged for this again. The page isn't just an interruption. It's a to-do item with an SLA.

📟
Primary responds, secondary backs them up, and the pager beats dinner but not health. The defining twist: the responder is an engineer, so every 3am page is pressure to make 3am quieter next month.

2The math of a sane rotation

Google's on-call playbook comes down to two numbers, and they're not negotiable in the way most "best practices" are. Break either and you don't get a slightly-worse rotation — you get a team that erodes.

The time number: nobody spends more than about 25% of their time on-call. Do the arithmetic and that means a single-site rotation needs at least 8 people (one on primary, one on secondary, so two shifts to cover, at a quarter each). Follow-the-sun across two sites lets you run 6 + 6 instead, because each site only covers its own daylight and nobody's woken at 3am at all.

The load number: at most about two incidents per 12-hour shift. Why two? Because handling an incident properly and writing the postmortem it deserves (Ch.7) takes hours. Past two, you're triaging in a hurry, the postmortems get skipped, and the lessons — the entire point — evaporate. It's a quality bound, not a comfort one. (And as section 6 warns: below the floor is its own kind of trouble.)

≤ 25%
of any one person's time on-call → ≥ 8 per rotation (or 6+6 across two sites)
≤ 2
incidents per 12h shift → enough time to handle each and write the postmortem
Interactive · rotation calculator Size the team · watch the two limits
team size 6 people
sites
incidents / week 7 / week
on-call time per person
incidents per 12h shift
sustainable

3Your brain on pages

Here's what nobody tells you when they hand you the pager: adrenaline makes you a worse engineer. Under acute stress your body does something useful for fighting a bear and disastrous for debugging a distributed system — it trades careful reasoning for fast reflexes. You stop deliberating and start pattern-matching: grabbing the first explanation that feels familiar and running with it.

That's brilliant when the problem really is the one you've seen before. It's confidently, catastrophically wrong on a novel one — and the 3am pages are disproportionately the novel ones, because the familiar failures already got automated away. So the responder most likely to guess is facing exactly the situation where guessing fails.

This is why procedures exist, and why a checklist isn't bureaucratic cowardice. A playbook is a prosthetic prefrontal cortex for 3am you — the careful reasoning you can't do under adrenaline, written down in advance by someone who could. Write your playbooks at 3pm, when you're smart, rested, and not currently on fire. 3am you will follow them gratefully.

🧠
Stress swaps reasoning for intuition — fast and great on familiar problems, confidently wrong on new ones. The 3am pages skew novel. So you borrow 3pm-you's judgment in the form of a written procedure.

4Mitigate first, diagnose later

The single most natural instinct during an incident is also the wrong first move. Your brain wants to understand the problem — what broke, why, how. Resist it. While you're reading stack traces, users are still getting errors.

First job: stop the bleeding. Roll back the deploy that started it (Ch.5). Drain the region that's misbehaving. Shed load. Flip the feature flag off. None of these tells you what went wrong — and that's fine, because explaining the outage is not your job right now. Ending it is.

Root cause is a daytime activity. You collect what you need to investigate later — logs, graphs, the deploy SHA — and you go back to bed with the service healthy. The on-caller's success metric during an incident is "the page cleared," not "I figured it out." A clever diagnosis that arrives after an hour of downtime is a worse outcome than a dumb rollback that arrives in ninety seconds.

Interactive · first move trainer 0 / 3
SEV2
page 1 / 3

5Playbooks and escalation

A page that says "checkout is down" hands you a blank page at the worst possible moment. A playbook turns that blank page into a procedure: here's what to check first, here's what to try, here's who to call if it doesn't work. It won't cover the truly novel outage — but it gets you past the panic and into motion, and motion is what 3am you is short on.

The other skill nobody teaches: escalation is not failure. Pulling in a second brain — the secondary, a teammate, the service owner — early is how grown-up teams operate. The lone hero debugging in the dark for four hours isn't admirable; it's a postmortem finding (Ch.7), the line that reads "responder did not escalate." Waking someone up to help is cheaper than every extra minute of downtime. Escalate early, escalate often, and stop treating it as a confession.

💡
Escalation encoded as data, not as a habit: policy(Sev1) says page primary now, secondary at +5m, the incident commander at +15m. 3am you doesn't have to remember the chain — and like the alerts in Ch.4, a policy that's data is one you can review in daylight.

6Too quiet is also a problem

You'd think the dream rotation is one that never pages. It isn't. An on-caller who hasn't been woken in months has forgotten how the machinery works — where the runbooks live, how to drain a region, what the rollback command even is. When the rare genuine disaster finally lands, they're rusty exactly when it counts. It's the irony of automation from Chapter 3: automate everything away and the humans lose the muscle for the times the automation can't help.

Google's fix is the same one fighter pilots and surgeons use — deliberate practice. Two forms:

  • Wheel of Misfortune — a role-play game. A game master replays a real past incident; the on-caller has to respond out loud, in real time, while the team watches. Cheap, frequent, and it keeps the procedures warm.
  • DiRT-style disaster exercises — failures injected on purpose, during business hours, with everyone watching. You break it deliberately, in daylight, with the smart 3pm version of the team in the room — so the broken-at-3am version has done it before.

The logic is identical to the load floor in section 2: too few real incidents, and you manufacture safe ones, on your terms.

Interactive · wheel of misfortune Spin · then reveal the response, step by step
Spin the wheel to draw a disaster.

7Sustainability

On-call is a cost the company is paying with your nights and weekends, and a sane org accounts for it like any other cost. A rough shift — the kind with three pages and a 4am call — earns time off, no questions, no guilt. On-call pays, in money or in time, because uncompensated 3am labor is just slow attrition.

And incident load is tracked like any other metric. When it climbs past what the team can absorb, the response isn't "work harder" — it's structural: push work back to the dev team, or hand the pager back entirely until the service is fixed. It's the same lever as the 50% toil cap from Chapter 3: a hard ceiling that converts overload into a forcing function instead of a death march.

Underneath all of it is one claim that organizations love to forget: burnout is a reliability risk, not an HR footnote. A burned-out on-caller responds slower, reasons worse, learns nothing from the incidents they barely survive, and eventually quits — taking all their hard-won context with them. Protecting the humans on the pager isn't kindness you do despite reliability. It is reliability.

🛟
Rough shifts earn rest, on-call gets paid, load is a tracked metric with a hard cap, and burnout is treated as the reliability threat it is. The pager is a shared resource — spend it like one.

8Check yourself

3 questions · instant feedback 0 / 3