The outage already happened — you've paid the tuition, and the postmortem is how you collect the lesson. But the moment analysis turns into blame, people start hiding facts, and the lesson is gone.
An outage is the most expensive teacher you'll ever hire. It charges in revenue, in error budget (Ch.2), in the sleep of whoever was on-call (Ch.6). The bill is non-negotiable; the only choice you get is whether you walk away having learned something.
The thing that decides that is the postmortem — a written, blameless account of what broke and why. Get it right and the outage becomes a permanent upgrade to the system. Get it wrong — turn it into a hunt for the guilty — and you pay the bill twice: once for the outage, and again when people quietly stop telling you the truth.
This chapter is about writing the receipt. We'll cover why blameless isn't a nicety but a data-integrity strategy, what goes in the document, why root causes always come in plural, and why a postmortem with no tracked follow-up is just expensive theater.
Every outage costs something real. It burns revenue while checkout is down. It drains the error budget you were saving for launches (Ch.2). It spends the sleep and goodwill of the on-caller who got paged at 02:47 (Ch.6). That money is gone the moment the incident starts — you can't get it back.
What you can get back is the lesson. The postmortem is the receipt: written proof that you got something in exchange for what you paid. It says, in durable text, "here is what broke, here is why, here is what we changed so it can't break that way again." That last clause is the whole point — it converts a one-time loss into a permanent fix.
Skip it, and you've paid full price for a lesson and left it on the counter. Worse: the next person to hit the same trap pays the same tuition all over again, because nobody wrote down where the trap was. An incident without a postmortem is just a bad day you'll get to have again.
Here's the failure mode that feels like success. The investigation reaches "Bob pushed the bad config," everyone nods, Bob looks at his shoes, and the meeting ends. It feels resolved. It has fixed nothing — because the next Bob can push the next bad config tomorrow, through the exact same door.
Humans erring is a constant, like gravity. You will never ship a release that removes it. So "a human made a mistake" is never a root cause; it's the weather. The actual bug is a system in which one tired, unreviewed keystroke can reach all the way into production and take it down. Fix the human and you've fixed one instance. Fix the system and you've fixed the whole class.
There's a second, sneakier cost to blame, and it's the one that really hurts. Punish people for incidents and they will, very rationally, stop telling you about incidents — the near-misses, the "huh, that was weird," the small saves that are your richest data. Blameless culture isn't kindness for its own sake. It's how you keep your data supply honest. Scare the witnesses and you go blind.
You do not want to be deciding "is this postmortem-worthy?" at 4am, exhausted, with a manager asking when you'll be done. So you decide the triggers in advance — the same move as the error-budget policy in Ch.2. Write the list once, while calm, and then it's not a negotiation; it's a checklist.
A reasonable trigger list looks like this:
The point of a fixed list is that it removes ego from the decision. Nobody is being singled out; the criteria fired. A postmortem is a normal, expected, slightly boring response to a class of event — which is exactly how you want it to feel.
A good postmortem has a predictable skeleton. Predictable is a feature — the reader knows where to look, and the writer can't quietly skip the uncomfortable parts. The pieces:
Then the part everyone underrates: the trigger-versus-root-cause distinction. The trigger is the thing that happened right before the boom. The root cause is the loaded condition that made the trigger lethal. Confuse them and you'll "fix" the trigger and leave the gun loaded.
And finally, action items — each with an owner and a deadline. Not "we should improve validation" (nobody, never) but "@priya adds schema validation to the config pipeline, due Friday." We'll come back to why these are the only part that actually matters in section 7.
Big outages are never one mistake. A single bad keystroke shouldn't be able to take down production — and in a healthy system, it can't, because there are layers of defense in the way: code review, tests, a canary, alerting, rate limits. For a real outage to happen, several of those defenses have to fail at once.
Picture each defense as a slice of Swiss cheese. Each slice has holes — gaps where it doesn't catch a given problem. On most days the holes don't line up: a problem slips through review's hole but gets caught by the canary. The outage is the one day all the holes happen to line up and a clean tunnel opens from "oops" to "production is down." This is the famous Swiss cheese model.
So "find THE root cause" is the wrong quest — it sends you looking for one villain when there were four open holes. The right question is: which holes lined up, and which one is cheapest to close? You rarely need to plug all of them. Close any single hole on the tunnel and that exact outage can't recur — pick the cheapest plug and move on.
Read a timeline after the fact and the story writes itself. Every clue glows neon. "The error rate ticked up at 02:31 — they should have known." Of course they should have. You have the answer key open in front of you. The smugness is automatic, and it's poison.
Because the on-caller at 02:47 had no answer key. They had three dashboards, two of which were lying to them, a fresh page, and no narrator whispering "the database, it's the database." The clue that looks neon to you was one flicker among forty, half of them red herrings, with no label saying THIS ONE MATTERS.
So the discipline is simple and hard: judge every decision only against what was knowable at that moment. Not what you know now. Anything else is hindsight bias, and it wrecks two things at once — it's unfair to the responder, and it teaches you the wrong lesson, because "they should have known" prescribes no fix you can actually build.
Here's the brutal test of whether a postmortem was real. A few weeks later, did anything change? A postmortem with no tracked follow-through is theater: cathartic, well-attended, beautifully written, and completely useless. Everyone felt better and the gun stayed loaded.
Action items go where work goes: the bug tracker, with owners and priorities, the same as any other engineering task. Closing them is real engineering — and it counts against Ch.3's 50% project-work budget, because that's exactly what it is. "Add validation to the config pipeline" is a feature; treat it like one.
Then do the cultural part, which is what makes the whole thing self-sustaining: share postmortems widely so the whole org learns from one team's outage, and reward the well-written ones. At Google a good postmortem is a flex, not a confession — you wrote the document everyone else now learns from. Make it shameful and people write the shortest legal-cover document they can. Make it admired and people write the truth.
Postmortems are just data. Model them as types and the follow-through becomes a query you can run in a meeting:
rootCauses: List[Cause] makes "the one root cause" un-representable, and overdue is the query you run in standup — if it returns a non-empty list, the last outage isn't really closed yet.