Alert on symptoms, not causes

"Monitoring" is one word doing four different jobs, and almost every monitoring disaster comes from confusing them. Sort them out and the rest of the chapter is downhill.

Only the last one reaches into someone's evening and ruins it. The other three are for eyeballs, not pagers. The classic failure mode: a graph that belongs on a dashboard gets wired to the pager because "we should know if this happens" — and now you're paging on something nobody can act on at 3am.

Here's the single most useful distinction in the whole chapter. Your user has no idea your disk is full. They know "checkout is slow" and "my upload failed." Those are symptoms — the things a human on the other end actually experiences.

A cause is your best guess about why: disk full, GC thrashing, a replica lagging, a dependency timing out. Causes are hypotheses. Symptoms are facts. And here's the punchline: page on symptoms, because they're true by definition. If checkout is slow, something is wrong, full stop — you don't need to know what yet.

Keep the causes on dashboards, ready for the human the symptom just woke up. The classic trap is the opposite: someone tries to page on every possible cause — every disk, every queue, every lag metric — and builds a wall of alerts, most of which hurt nobody. Think of it like exceptions: you catch and surface the failure the caller can feel; you don't throw a separate alarm for every line of the stack trace.

Two ways to watch a service, and you want both. Black-box monitoring probes from the outside, exactly like a user would: hit the login page, run the checkout flow, see if it works. It catches everything that matters — including the failures you never thought to instrument — and it explains absolutely nothing. It's a smoke alarm: it tells you there's fire, not where.

White-box monitoring reads the internals: per-component metrics, queue depths, error counters, the saturation of every resource. It explains everything and predicts trouble — but it only sees what you thought to measure. The disk you forgot to track fills up silently.

The deal between them is clean: black-box pages you (a user-visible symptom is failing), and white-box tells the person who got paged where to look. One detects, the other localizes. Either one alone is half a monitoring system.

You met them in Chapter 1: latency, traffic, errors, saturation. Here's their real job — they're the symptom vocabulary. Between the four of them, you can express almost every user-visible failure there is. That's why they're the things you page on.

• Latency — slow is a symptom. Caveat worth repeating until it's reflex: track failed-request latency separately. A fast error and a slow success look identical to an average, so a flood of instant 500s can make your latency graph look great while the service is on fire.
• Traffic — demand. The context that tells you whether a spike in everything else is your fault or the world's.
• Errors — failing is a symptom, whether loud (500s) or quiet (a wrong answer served fast).
• Saturation — the special one. It's the only signal that predicts. "Disk full in 4 hours" is a cause, yes — but it's a cause that's about to become a symptom on a clock you can see. That's the rare cause worth paging on: not because it hurts now, but because it provably will, and soon.

So the rule "page on symptoms" has exactly one principled exception: a saturation trend with a deadline. Everything else cause-shaped stays on the dashboard.

Here's the test every alert has to pass. A page means one thing: a human must think, right now. Not look, not acknowledge — think. Judgment a machine can't supply.

So run the triage. If the response is mechanical — "restart it," "clear the queue," "fail over" — then it's not a page, it's a script you haven't written yet (hello, Chapter 3). Automate it. If it can wait until morning, it's a ticket, not a page. Only what's left — user-visible, happening now, and needing a human's judgment — earns the right to wake someone.

Why so strict? Because every non-actionable page is a little dose of poison. It trains the on-caller to treat the pager as noise, and one ignored real page erases a hundred quiet nights. Pager fatigue isn't a character flaw or a willpower problem — it's a monitoring bug, and you fix it by deleting alerts, not by lecturing the human.

Look at the alerts that actually caught your last few real outages. They're embarrassingly simple: "error rate above X for Y minutes," "probe failing," "burn rate too high." The clever ones — the multi-condition, machine-learned, six-way-correlated rules someone was very proud of — those are the ones that rot, fire on the wrong things, and that nobody dares touch.

So prune, on purpose, like you'd prune dead code. Delete any alert that hasn't fired meaningfully in a quarter (if it never fires, it's not protecting you; if it fires and you ignore it, it's noise). Delete any rule nobody can explain at 3am. Delete any threshold someone tuned by superstition and can't justify.

The reframe that makes this stick: your monitoring config is code. It goes through review. It has an owner. And the healthiest thing you do to it most weeks is delete from it.

Put it all together. A well-monitored service looks like this — and it's calmer than you'd expect:

• A handful of symptom-based page alerts on the golden four — or on burn rate (Chapter 2), which is just the golden four with a deadline attached.
• Cause-based dashboards behind them — every disk, queue, and lag metric you might want, sitting one click away for the human the page woke up.
• Black-box probes on the critical user journeys — login, checkout, search — so you find out before your users tweet.
• A pager that fires rarely enough that everyone still respects it. That last one isn't a nice-to-have; it's the whole point. A respected pager is the only kind that works.

If your monitoring has those four things, you've built the rare system that wakes a human only when a human is genuinely the answer. Everything else in this chapter was just how to get here.