Overfitting:
the danger of thinking too much

You've got ten data points. You fit a curve through them — and not just close to them, through them. Every single point, dead centre. Your error is zero. Not "small": zero. You would sign off on this model in a code review.

Then a new point arrives, from exactly the same source as the first ten. Your beautiful curve misses it by a mile. Not slightly — it misses worse than a straight line drawn by someone who wasn't paying attention would have.

What went wrong? Nothing, technically. The model did precisely what you asked. You asked it to minimise error on the data you showed it, and it found a way: it bent itself into whatever shape it needed to reach each point. But those points weren't pure signal. Each one was the real pattern plus a little bit of measurement slop, luck, and noise. And a model flexible enough to reach every point is a model that has faithfully learned the slop too.

That's the whole idea, and it deserves its name: overfitting. You didn't learn the pattern. You memorised the noise — and noise, by definition, does not repeat.

Here's the deeper reason this keeps happening, and it isn't really about curves.

You never optimise the thing you actually want. You can't — the thing you want is usually diffuse, slow, and impossible to put a number on. So you optimise a measurement of it. Grades stand in for learning. Revenue stands in for value delivered. Steps stand in for health. Story points stand in for progress. Test coverage stands in for correctness.

Every one of those measurements is a decent model of the real goal — in the range where you found it. Which is exactly the situation from the last section: a model, fitted to a limited slice of the world, that works fine until you push it somewhere it's never been. And optimisation is pushing it somewhere it's never been. That's its whole job.

So: a Soviet nail factory judged on the number of nails produces millions of tiny useless tacks. Judged on tonnes of nails, it produces one gigantic useless nail. A military that counts enemy casualties as its measure of progress gets a body count, and loses the war the body count was supposed to represent. Colonial Delhi offered a bounty per dead cobra, and got cobra farms; when the bounty was cancelled the farmers released their stock, and the city ended up with more cobras than it started with.

None of these are stories about stupid or wicked people. They're stories about people doing precisely what they were measured on. The rule is short and it does not care how you feel about it: whatever you measure, you get more of — including the parts you didn't mean.

Once you have the shape in your head — a model beautifully fitted to conditions that no longer hold — you start finding it everywhere, and most of the sightings aren't about computers at all.

• Your appetite. A powerful craving for fat, salt and sugar was an outstanding model of a world where calories were scarce and seasonal and you had to catch them. It scored beautifully on that training set. It is catastrophically overfit to a world with a supermarket on the corner. Your taste buds aren't broken; they're fitted to different data.
• Superstition. Lucky socks, a lucky mug, the exact order you tap the desk before a deploy. Ten data points, one of them a win, and a brain that is exquisitely good at finding patterns fits a curve through the noise. Rituals are overfitting with a ceremony attached.
• The immune system. A classifier trained on real threats, that in a too-clean environment starts flagging pollen and peanuts as hostile. Allergies are false positives from a model with more capacity than its training data can justify.
• Training for the drill instead of the fight. Bill Belichick's line is "practice how you play," and it's a technical point, not a motivational one: whatever you rehearse is your training set. There's a much-repeated cautionary tale from firearms training — officers drilled to fire, catch their brass, and pocket it, because that's what tidy range practice required. The story goes that some were later found dead with spent casings in their hands. Whether or not the detail survives scrutiny, the shape of the lesson does: the drill quietly became the thing being optimised, and the drill is not the fight.

The common thread: none of these systems malfunctioned. Each one is excellent at the world it was fitted to. The world moved.

Here's the awkward part. From the inside, overfitting looks exactly like success. Your error is falling. Your fit is getting tighter. Every number on your dashboard is green. There is no signal, anywhere in the data you're looking at, that tells you you've crossed the line — because the data you're looking at is the data you fitted to, and on that data, wiggling always helps.

So the fix isn't cleverness. You cannot out-think this one. The fix is honesty, and it costs you something real: take a slice of your data and refuse to fit on it. Lock it in a drawer. Fit on the rest. Then, and only then, check the model against the slice you held back.

Now you have two numbers instead of one, and their disagreement is the entire signal. Training error slides steadily toward zero as your model gets more flexible — it always will, that's not information. Held-out error falls too, at first, because early flexibility is buying you real pattern. And then, at some point, it turns around and starts climbing. That turn is the line. Everything after it is memorised noise. Everything before it is pattern you hadn't captured yet.

Hold-one-slice-back has a name — cross-validation — and the industrial version just does it repeatedly: carve the data into k chunks, take turns holding each one out, average the results. Same idea, less luck.

There's a stronger move than holding back a slice, and it costs almost nothing: test against a different kind of data entirely.

Held-back data is still data of the same species, collected the same way, with the same blind spots baked in. If your measurement instrument is subtly wrong, holding some of its output in reserve won't save you — you'll validate a wrong model against a wrong yardstick and get a clean bill of health. What catches that is an unrelated measurement, ideally a scrappier one.

• Hiring. Interviews are one instrument, and candidates optimise against it — hard, and rationally, because that's what's measured. So interview them and look at work they've actually shipped. The two measurements have almost nothing in common, which is exactly what makes disagreement between them informative.
• Admissions and grades. A transcript is a model of a student fitted by the same institutions that taught the test. An essay, a portfolio, a conversation — different instruments, different failure modes, no shared blind spot.
• Your own metrics. Your engagement numbers are up 30%. Go and watch five people use the thing. If the crisp quantitative story and the scrappy qualitative one disagree, the disagreement is worth more than either number alone.

The test is simple, and you can apply it to any model, including the ones in your head: if it only survives on the data it was born on, it's memorising.

Cross-validation tells you that you overfit. It doesn't stop you. For that, there are two moves, and they turn out to be the same move.

Move one: charge yourself rent. Every extra factor you add to a model, every extra term, every extra "well, it also depends on…" — make it cost something. Then a factor can only join if it buys back more error than it costs in rent. Weak factors, the ones that were only ever explaining noise, can't pay. They get evicted. You're no longer asking "does this help?" (everything helps, on training data). You're asking "does this help enough to be worth it?" — a question with a real answer.

That's regularization, and the specific version worth knowing by name is the Lasso. Its charge has a lovely property: it doesn't just shrink weak factors toward zero, it shoves them to exactly zero. Not 0.003 — gone. Turn the rent up and what you're left with is a short, honest list of the factors that could actually pay, and a model you can hold in your head.

Move two: prefer the simpler story. Given two explanations that fit the evidence equally well, take the simpler one. That's Occam's razor, and it's usually taught as a matter of taste or elegance. It isn't. It's Chapter 6 in disguise: a razor is a prior. Saying "complicated explanations are less likely to be true" is exactly the same statement as "complicated explanations carry a penalty," which is exactly the same statement as "here is a prior that puts less probability mass on complicated hypotheses." Prior and penalty are the same object wearing different hats — one in the language of belief, one in the language of arithmetic.

Now suppose you can't do any of that. No held-out slice, no penalty term, no λ to tune — just you, a decision, and no clean way to price complexity. There's still a move, and it's almost insultingly simple.

Stop earlier. A model that trains for less time is a model that has had less opportunity to overfit. It never gets flexible enough to reach the noise. In machine learning this is a technique with a name — early stopping — and it's deployed on purpose, not as a compromise. You watch held-out error, and the moment it turns upward you pull the plug, mid-training, and ship what you have.

Which reframes something you probably file under "constraints I'm stuck with". A tight deadline. A one-page limit. A budget that won't stretch. A meeting that ends at 3pm whether you're done or not. A rule of thumb instead of a proper analysis. Every one of those is a regularizer. It's not stopping you from finding the truth — it's stopping you from wandering past it.

The patron saint of this idea is Charles Darwin, who in 1838 faced the biggest decision of his life and handled it like a man with an excellent internal regularizer. He took a scrap of paper, drew a line down the middle, wrote "Marry" on one side and "Not Marry" on the other, filled in both columns, and gave the whole thing about two days.

Two columns, a handful of bullets each, one of which is "better than a dog anyhow". At the bottom he wrote "Marry—Marry—Marry Q.E.D." and, essentially, got on with it. He married Emma Wedgwood the following January and stayed married for forty-three years.

Now imagine the alternative. Darwin spends six months on this. He builds a weighted scoring model. He interviews married friends and gathers a dataset. He discovers that clever-men-at-clubs correlates negatively with long-run happiness at p < 0.05, in a sample of eleven. He would have produced something far more impressive, far more defensible, and considerably worse — because the evidence available to him about his own future was two paragraphs of noisy introspection, and no amount of machinery makes two paragraphs of noise say more than they say. His scrap of paper wasn't a shortcut around the analysis. Given his data, it was the right-sized analysis.

Time for the honest answer, because "sometimes simpler is better" is a fortune cookie and you deserve the actual rule.

How much complexity you can afford depends entirely on the quality of your evidence. Complexity is a purchase, and your data is the money. Clean, plentiful, stable data? Buy the elaborate model — you can afford it, and it'll pay you back. Noisy, scarce data about an uncertain future? Every extra factor you add is far more likely to be fitting the noise than the pattern, because in that data there is proportionally more noise than pattern to fit. You are not being rigorous. You are being precisely wrong.

So the rule, in one line: the messier your data and the more uncertain the future, the less you should let your model wiggle.

Which is uncomfortable, because it inverts where we usually reach for rigour. The moments that feel like they demand the most careful analysis — the high-stakes call, the murky situation, the decision that could go either way — are exactly the moments when the data is thinnest and the future least predictable. They're the moments you can least afford a complicated model. The elaborate spreadsheet for the hardest decision of your life is not rigour. It's a very expensive way to fit noise, with a bibliography.

And there's a last, practical twist. A simple rule has one more thing going for it that never shows up in the maths: you'll actually follow it. An elaborate policy you abandon in week three scores zero. A crude heuristic you keep for a decade compounds. Under real uncertainty — which is most of life — the simple rule you can execute beats the sophisticated one you can't, and it isn't close.

The whole chapter fits in one small function. Split the data. Fit on one half. Score on the other — and add a charge for complexity while you're at it. Then pick the model with the best score. That's it; the rest is fitting machinery you can swap out. Flip between the languages — the shape is identical.

This chapter said: when the problem is murky, don't solve it too hard. The next one goes further, and it sounds like cheating. Faced with a problem that's genuinely impossible to solve exactly, you're allowed to delete the constraint that's making it impossible, solve the easy fake problem instead, and then argue — rigorously, with a bound — about how close the fake answer must be to the real one. It's the most useful intellectual dodge in computer science, and it has a name.