Some day more traffic will arrive than your service can handle — a launch, a celebrity tweet, your own clients retrying in a panic. The services that survive are the ones designed to refuse some work gracefully instead of failing all of it heroically.
Every other chapter so far has been about steady state — measure it, budget it, automate it, learn from it. This one is about the day the steady state breaks: demand wins. More requests show up than you have machines to serve.
The naive instinct is to try harder — accept everything, queue it all, and let the servers grind until they melt. That's the heroic failure mode, and it takes the whole service down with it. The SRE instinct is the opposite and faintly un-American: decide who you're going to fail, fail them fast and cheap, and keep everyone else alive.
This chapter is a tour of the machinery that lets you do that without making things worse. The villains: a brutal latency curve, panicking clients, synchronized crowds, and feedback loops that turn one dead server into a dead cluster. The heroes: load shedding, graceful degradation, backoff, jitter, circuit breakers, and quotas.
You can push the day off. Buy more machines, forecast your growth, provision ahead of demand — that's capacity planning, and Chapter 10 is all about doing it well. But capacity planning only delays overload; it never prevents it. Forecasts are wrong, launches go viral, and someone always writes a client that retries in a tight loop. Sooner or later demand spikes past whatever you provisioned.
So the goal quietly flips. It's not "never be overloaded" — that's a promise you can't keep. It's "be overloaded without falling over." A service that gracefully sheds the excess at 130% load is healthy. A service that tips over and serves nobody at 101% is broken, no matter how pretty its dashboards look at 99%.
Which is why a service's real capacity isn't measured at its sticker number. It's measured by what happens at 110% of it. Push it just past the line and watch: does it degrade, or does it detonate?
Here's the cruelest curve in systems: latency versus utilization. It's flat, flat, flat — and then a wall. Your p99 sits at a comfortable 40ms from 10% load all the way to 70%, and you think you've got tons of headroom. You don't. Push to 90% and the same request takes 400ms. Push to 95% and it's a timeout.
The plain-English reason is queueing. When your servers are mostly idle, a request walks straight up and gets served. When they're mostly busy, every new request has to wait behind whatever's already running — and the wait compounds, because the thing it's waiting on is itself waiting. Past roughly 70–80% utilization, response times stop creeping and start exploding: each extra percent of load adds dramatically more delay than the last.
The formal name is queueing theory (the textbook toy is the M/M/1 queue, where latency scales like 1 / (1 − ρ) and ρ is utilization — as ρ approaches 1, latency approaches infinity). The practical lesson: running "efficiently hot" at 95% to save on machines means living one traffic burp away from the cliff. The headroom you deleted was the headroom that kept you off the wall.
Once you accept that overload will happen, the question stops being "how do I serve everyone?" and becomes "who do I fail, and how cheaply?" Load shedding is the answer: when more work arrives than you can do well, reject the excess early — at the front door, with a fast, cheap error — so that the requests you do accept get served properly.
The arithmetic is blunt. Serving 90% of your users a fast, correct response and handing the other 10% a clean "try again in a moment" beats serving 100% of your users a page that spins for thirty seconds and then dies anyway. Worse, the slow-then-dead path costs you real resources for every doomed request — you pay full freight to disappoint everyone. Degraded on purpose beats down by surprise.
The key word is early. A rejection is only cheap if it happens before the request has eaten a database connection, a thread, and forty milliseconds of CPU. Shed at the edge — a fast 429 Too Many Requests — not deep in the stack after you've already paid for the work you're about to throw away.
Shedding is binary: serve or reject. Graceful degradation is the subtler move — keep saying yes, but give a cheaper answer. Serve the slightly-stale cached value instead of recomputing it fresh. Return 10 search results instead of 100. Skip the "you might also like" panel that fans out to six backends. Drop the personalization and serve the generic page.
The user barely notices — they got a page, it was fast, the world kept turning. Your database absolutely notices: you just spared it the expensive query it was about to choke on. Most "full" experiences have a long tail of nice-to-haves that cost a fortune and add little; degradation is the art of dropping them under pressure and nobody filing a bug.
The catch — and it's the same catch that haunts Chapter 9's backups: an untested degraded mode is a second outage waiting inside the first. The cheap path you wrote two years ago and never exercised will have rotted: the stale-cache branch throws a null, the fallback endpoint was decommissioned. Build the cheap path before the emergency, and run it on purpose — regularly, in production — so that when you reach for it under fire, it's actually there.
Here's how a 10-second blip becomes an hour-long outage. A backend hiccups. Every client that got an error does the "helpful" thing and retries. Now your load has doubled — at the exact moment your backend was least able to handle it. The retries fail too, so they retry again. The hiccup you'd have shrugged off becomes a heart attack, and it's your own clients holding the defibrillator backwards.
This is a retry storm, and it's the single most common way a small problem becomes a big one. The cruel part: every client was behaving "correctly" in isolation. Correctness in a distributed system is a property of the crowd, not the individual.
The civilized rules that keep retries from amplifying the disaster:
delay backs off exponentially, caps the wait, then picks a random point in [0, exp] — full jitter, so no two clients march in step. mayRetry is the budget check: if your retries are already adding more than their share of load, you stop. A polite client is a distributed-systems citizen.Retries are clients panicking out of sync would be fine — the trouble is they synchronize. And synchronized clients are a self-inflicted DDoS, even when nothing's wrong. Every cron job fires at :00. Every cache entry you set with the same TTL expires in the same second, and a thousand requests stampede the database at once to refill it. Every client that dropped during your outage reconnects the instant you come back — and the recovery spike re-kills the patient you just revived.
This is the thundering herd: a crowd that moves as one. It's not malicious and it's not a bug in any single component; it's an emergent property of everybody doing the same reasonable thing at the same instant.
The fix is deeply unglamorous and almost always the same word: jitter everything. Spread cron jobs across the minute instead of firing them all at :00. Randomize cache TTLs by ±10% so they don't all expire together. Add random reconnect delays so the recovery wave arrives as a ramp, not a wall. The herd is dangerous only because it's synchronized; a little randomness desynchronizes it, and the stampede becomes a stroll.
Now the nightmare scenario, where overload becomes a feedback loop. One server in your fleet of five dies — maybe it OOM'd, maybe it just got unlucky. Its traffic doesn't vanish; the load balancer fans it out to the survivors. But the survivors were already running hot, so now they're running hotter. One of them tips over. Its load lands on the four three two that remain. Repeat.
This is cascading failure, and the horror of it is the math: each death makes the next death more likely, not less. It's a positive feedback loop, where "positive" means catastrophic — the system accelerates into the ground. By the time you're paged, the cluster is already dark, and naively restarting servers just feeds them straight back into the same overload that killed them.
The defenses are about breaking the loop before it runs away. Circuit breakers: cap the in-flight requests each server will accept and shed the rest, so an overloaded box stops accepting work instead of dying with it. Fail fast toward dependencies: if a backend is timing out, stop calling it for a bit rather than piling up threads waiting on it. And the genuinely counterintuitive one — sometimes you deliberately drop a chunk of traffic, on purpose, to let the cluster catch its breath and climb back off the cliff. You shed load to save load.
Last truth, and it's the multi-tenant one. Your service has many clients, and they share one pool of capacity. Which means one badly-behaved client — a runaway batch job, a buggy retry loop, a customer who 100×'d their traffic overnight — can eat everyone's capacity and starve the well-behaved neighbors. Shared capacity with no fences is a tragedy of the commons waiting to happen.
The fence is a per-client quota: each tenant gets a guaranteed slice and gets throttled when they exceed it, so one client's bad day stays one client's bad day. Good fences make good neighbors, and in distributed systems the fences are quotas.
The second tool is criticality: tag every request with how much it matters. A checkout outranks a prefetch. A user-facing read outranks a background reindex. Then — and this is where it all comes together — when you shed load (section 3), you shed the low-criticality tiers first. The prefetches and the batch jobs get the 429; the checkouts sail through. Your load shedding now degrades the experience (a slower prefetch, a delayed report) long before it ever touches the business (a failed purchase). That's the whole chapter in one sentence: refuse the work that matters least, so the work that matters most always gets done.