Site Reliability Engineering · ch.3 · eliminating toil
📟 Chapter 3 · The 50% rule and the war on repetitive work

The boredom is a signal

If a task bores you, that's your engineer brain telling you a script should be doing it. This chapter is about the work that keeps services running without making them better — how to recognize it, why it's poison, and how to delete it.

Every engineer knows the feeling: you've restarted the same stuck job four Tuesdays running, and somewhere in the back of your skull a voice says this is beneath me. That voice is correct, and it has a name.

The name is toil, and SRE treats it as the enemy. Not because it's hard — it's the opposite of hard, it's mind-numbing — but because it's a tax that grows with your success, it caps your career, and it quietly turns a team of engineers into an ops team wearing engineer name-tags. Chapter 1 told you SRE exists to stop that. This chapter is the actual war plan.

We'll pin down what toil is (and the look-alikes it isn't), why it rots a team from the inside, the hard 50% budget that holds it back, and the migration path from a panicked human at 3am to a system that quietly fixes itself.

1What toil is (and isn't)

Toil isn't "work I don't like." Plenty of important work is unpleasant. Toil is a specific, recognizable kind of work, and it ticks most of six boxes at once:

  • It's manual — a human types the commands.
  • It repeats — you've done it before, you'll do it again, same steps.
  • A machine could do it — there's no judgment a script couldn't encode.
  • It's reactive, not strategic — you're responding to a page, not building toward anything.
  • It leaves nothing behind — when you're done, the service is exactly as good as before, just unbroken again.
  • It grows with the service — twice the fleet, twice the disks to clear.

Restarting the same stuck job every Tuesday: six for six. Pure toil. Now the look-alikes, because mislabeling them wastes everyone's time:

Meetings and email are annoying and they eat your week, but they're overhead — the cost of belonging to a company, not the cost of running a service. Hard design work — sketching next year's capacity plan, building the very automation that kills the toil — is difficult and reactive-feeling some days, but it builds something. It leaves the service better than it found it. Difficulty isn't the test. The test is whether anything durable survives the work.

Interactive · the toil scorer Toggle the traits · or load an example
load:
Flip some chips, or load an example. The dial swings from engineering toward pure toil as the boxes light up.

2Why toil is poison

If toil were merely boring, you could just grit your teeth. It's worse than boring — it's corrosive, in four ways that feed each other.

Careers stall. Nobody gets promoted for hand-clearing disks. The promo packet wants impact that outlives the quarter, and toil leaves nothing to point at. Your most experienced people spend their best years doing the same repairs an intern could do.

Morale rots. Smart people who took the job to build things, doing chores instead, start polishing their résumés. The good ones leave first.

Velocity drops. Every hour on the treadmill is an hour not spent making the next thing automatic. Toil is time you can never invest.

Toil breeds toil. This is the nasty one. Every manual workaround becomes load-bearing — someone scripts a fragile fix that needs babysitting, which becomes its own recurring chore, which spawns the next one. The mess compounds.

Add it up and you watch a team of engineers quietly become an ops team with engineer salaries — running the service by hand, too buried to automate, too senior to be cheap. That is the exact failure mode SRE was invented to prevent (Chapter 1's whole premise). Toil is how you slide back into it without ever deciding to.

☠️
The slow death has no alarm. No single Tuesday's disk-clearing is a crisis — that's why it works. You only notice once the team's calendar is full of chores and the roadmap hasn't moved in two quarters.

3The 50% line

SRE's defense is a number, the same way the error budget was a number. The rule: every SRE spends at least half their time on engineering that has lasting value. Toil — manual ops, ticket-grinding, on-call firefighting — is capped at the other half.

And it's measured, not vibed. Toil gets counted — surveys, ticket counts, time tracked against on-call. You can't manage what you won't measure, and "I feel pretty busy" is not a measurement.

The cap has teeth. If a team's toil overflows past 50%, that's not "work harder" — it's a signal the system is broken, and the fix flows back to the people who can change the system: work gets pushed to the dev team. More dev ownership of operations, up to and including handing the pager back. The threat is real, which is what makes it work.

Notice the shape: it's a budget, like the error budget. The number decides who does what, not the loudest engineer in the standup. A political fight becomes arithmetic — exactly the move from Chapter 1.

4The automation ladder

You don't go from "human does everything by hand" to "system heals itself" in one leap. You climb, and there are five rungs. Each one trades effort now for freedom later:

  • Rung 1 · no automation — a human and a prayer. Someone remembers the steps, or doesn't.
  • Rung 2 · a runbook — the human follows written steps. The knowledge escapes one person's head.
  • Rung 3 · a script the human triggers — the steps are code now; a person still pulls the lever.
  • Rung 4 · automation something else triggers — a monitor or a timer fires it; no human in the loop, but you wrote the trigger.
  • Rung 5 · the system heals itself — it detects, fixes, and verifies, and nobody is paged. The work has vanished.

You don't jump to rung 5. Each rung makes the next one cheaper and safer to build, because by then the steps are written down, then tested, then trusted. The ladder is the plan.

Interactive · the automation ladder Example: TLS certs expire monthly · click a rung
upfront cost
cost / occurrence
pages / year
hours / year
Pick a rung to see what TLS-cert renewal costs you at that level of automation.

5The compounding payoff

Automation is interest-bearing. Every task you automate frees the time to automate the next one, which frees more time, which… you've seen this curve before — it's the one that makes saving early beat saving a lot. Toil deleted today pays out every week for the rest of the service's life.

There's a catch, and it's the reason teams stay stuck. The quarter you spend writing the automation, you're slower — you're paying the build cost up front while still doing the chore by hand. That's the J-curve: things dip before they climb. A team drowning in toil looks at that dip and says "we can't afford to stop and automate" — which is precisely why they never escape. The busy-ness is self-perpetuating.

The way out is to treat the dip as an investment with a known payback period, not a luxury. The widget below lets you watch the math: a little time invested now bends the toil curve down hard later.

Interactive · the compounding machine Invest time now · watch 24 months of toil
% time invested in automation 20%
toil-hours @ month 24
hours saved vs 0%
break-even
At 0% the service grows and the toil grows with it — until it eats the whole week. Invest a slice and the curve bends.

6When NOT to automate

Automation is the answer to toil, but "automate everything" is its own kind of foolishness. Three cases where you should put the keyboard down:

One-offs. If you'll do it once, scripting it costs more than just doing it. The break-even math from the last section runs the other way — there's no second occurrence to amortize against. Do it by hand, write a note, move on.

Things still changing weekly. Automating a moving target means rewriting the automation every time the target moves. Wait for the process to settle before you carve it into code; a runbook is fine in the meantime.

The subtle one: automation that papers over a system you no longer understand. A script that "just works" can become a blindfold — it keeps the lights on while the humans forget how the thing actually behaves. Now you're one weird failure away from a team that can't operate its own service without the magic box.

Which is the irony of automation: the better it works, the rustier the humans get for the day it doesn't. The more reliably the box handles things, the less anyone remembers how. The answer isn't less automation — it's deliberately practicing the manual path, which is exactly what Chapter 6's disaster drills are for.

7Runbook → script → service

Here's the migration path in practice — the same climb up the ladder, but as a set of moves you can actually ship one at a time.

Write the runbook first. Prose forces precision. The moment you have to write "step 4: restart the worker" you discover the seven things you secretly knew but never said — which worker, in what order, how you check it came back. The runbook is where the tacit knowledge gets dragged into the light.

Turn each step into code. Code forces honesty in a way prose can't. A runbook can say "check if it looks weird"; a type checker will look you dead in the eye and ask weird how, exactly, and what's the type of "weird"? Every fuzzy step either becomes a real predicate or gets exposed as a step you never actually understood.

Move the trigger from a human to the system. Once the script is trusted, hand the lever to a monitor or a schedule. The human stops being the thing that notices and starts being the thing that gets told only when the automation gives up.

The beauty is that each stage is shippable on its own. A runbook beats tribal knowledge today. A script beats a runbook next week. Self-healing beats a script next quarter. You're never blocked waiting for the whole thing — you climb one rung, ship it, and climb again.

💡
The runbook said "restart it, then check if it looks weird." The for-comprehension over Either disagreed: every step must return Either[Escalate, _], so "looks weird" has to become a real check that either passes or pages a human. Vague steps don't typecheck — which is the whole point.

8Check yourself

3 questions · instant feedback 0 / 3