Every large system starts as one box that does everything, and grows by peeling off one job at a time the moment that box can't keep up. This chapter walks the whole arc — load balancer, replicas, cache, CDN, queue, shards — one bottleneck at a time.
System design has a reputation as a memory test — name the AWS service, recite the buzzword. It isn't. It's one move, repeated: find the part of your system that's drowning, and give that one job its own machinery.
We'll start with the smallest thing that can serve a web request: a single computer running your web server, your application code, and your database, all at once. It works. It works for longer than you'd think. And then, slowly, it stops.
Each section here is the same story told again at a bigger scale. Something becomes the bottleneck. We peel that responsibility off the single box and hand it to a dedicated layer — a load balancer, a read replica, a cache, an edge network, a queue, a shard. By the end you'll have walked from one server to an architecture that comfortably serves millions, and you'll know why each layer earned its place.
Picture one rented computer. On it runs the web server that speaks HTTP, the application code that holds your business logic, and the database that stores your data — all sharing the same CPU, memory, and disk. A user opens your site and a small relay race happens behind the scenes.
app.com into an IP address like 93.184.x.x.For a brand-new product this is exactly right. One box is cheap, simple to reason about, and easy to deploy to. You should not add a single thing to it until something actually hurts. The trouble is only that everything lives in one place: when traffic climbs, the web server, the app, and the database fight over the same resources — and if that box dies, the whole product is down. This is the architecture we spend the rest of the chapter dismantling, one layer at a time.
The first thing that fights for resources is almost always the database — it's memory-hungry and disk-bound, and it doesn't like sharing. So the first peel is simple: move the database onto its own machine. Now the web/app box and the database box can be tuned, sized, and restarted independently.
Once they're separate you face the question that runs through this whole chapter: when a box is overloaded, do you buy a bigger box or buy more boxes?
Vertical scaling buys you time; horizontal scaling buys you a future. Real systems use both: scale the database vertically for a while, scale the stateless app tier horizontally from early on.
To run more than one app box you need something to spread requests across them: a traffic cop sitting in front that hands each incoming request to whichever server is least busy and quietly skips any server that's down. That's a load balancer. Users only ever talk to the load balancer's address; the pool of app servers hides behind it.
But the moment you have many app servers, a sneaky bug appears. Say a user logs in and server A remembers "this person is Alice" in its own memory. The next request gets routed to server B — which has never heard of Alice. Suddenly she's logged out. The data a server keeps about a specific user between requests is called session state, and pinning it to one machine quietly breaks horizontal scaling.
The fix is to stop keeping that memory on the app servers. Push session state out into a shared store every server can read — a small fast database like Redis. Now each app server holds nothing personal; it reads what it needs, does the work, and forgets. Any request can land on any server and get the same answer. We call such a server stateless.
Keep a few up-to-date copies of the database. Send every write (create, update, delete) to one designated copy, and spread all the reads across the others. Because most apps read far more than they write — timelines, product pages, profiles are viewed thousands of times for every edit — this one move multiplies your read capacity enormously. The write-taking copy is the primary; the read-serving copies are replicas, and the technique is primary-replica replication.
It also buys resilience: if a replica dies, the others absorb its reads. If the primary dies, you promote a replica to be the new primary — a failover.
The catch is timing. A write lands on the primary and then has to travel to the replicas, and that trip isn't instant. For a brief window a replica is serving slightly old data. That delay is replication lag, and it causes a specific, infuriating bug: a user posts a comment (write → primary), the page reloads (read → a replica that hasn't caught up yet), and their own comment is missing. Demanding that you can immediately read what you just wrote is the read-after-write guarantee, and lag is what breaks it.
If a thousand users ask for the same thing, why compute the answer a thousand times? Keep the result somewhere very fast — in memory — and hand out the saved copy. The first request does the slow work; everyone after gets the cheap copy. That fast saved-answer store is a cache.
The common pattern is cache-aside: when a request comes in, check the cache first. Hit — return it, done, no database touched. Miss — fall through to the database, compute the answer, stash it in the cache, then return it. Each cached entry carries a TTL (time-to-live) so it expires and gets refreshed, and when the cache fills up it throws out whatever was least recently used (LRU eviction) to make room.
Two failure modes to respect. A cold cache (just started, or just flushed) means everything is a miss and the database briefly feels the full load. Worse is the thundering herd: a popular entry expires and a thousand simultaneous requests all miss at once, all stampede the database to recompute the same thing. And then there's invalidation — keeping the cached copy in sync with the real data when it changes is genuinely hard. As Phil Karlton put it, "there are only two hard things in computer science: cache invalidation and naming things."
Your server lives in one city. Your users live everywhere. A user in Sydney asking a server in Virginia for a 2 MB image is paying for light to cross an ocean and back — every single time. The fix is to keep copies of your unchanging files (images, video, CSS, JavaScript) in data centers scattered around the globe, and serve each user from the one nearest them. That global network of edge caches is a content delivery network, or CDN.
Why it works: those static assets are the same for everyone, so they cache beautifully, and the win is geographic — you trade a 300 ms round trip across the planet for a 20 ms hop across town. The first time an edge server is asked for a file it doesn't have, it fetches it from your origin and keeps it; this is an origin pull. After that, your origin barely sees the traffic.
You control how long edges hold a copy with cache-control headers (the asset's TTL), and the same hard problem from caching returns: when you ship a new version of a file, you must invalidate the old copies at the edge — usually by changing the filename (app.9f3c.js) so the new URL is simply a different, uncached object.
When a user uploads a photo, they shouldn't have to wait while you generate six thumbnail sizes, scan it, and email their followers. Take the order, hand back the receipt instantly, and do the slow work in the background. The way you hand work off is to drop a little "please process this" message onto a list, and have separate machines pull messages off that list and do the work. The list is a message queue; the machines draining it are workers.
The web tier becomes a producer — it just enqueues jobs and returns. The workers are consumers — they dequeue and process at their own pace. This decouples the two: the web tier stays fast and responsive no matter how slow the background work is. And it smooths spikes beautifully — if a million uploads arrive in a minute, the queue absorbs the burst and the workers chew through it steadily instead of falling over. Need to go faster? Add more workers; they're stateless too.
Replicas multiply your reads, but every replica still holds the entire dataset and every write still goes through one primary. Eventually the data won't fit on one machine, or the write rate overwhelms a single primary. The answer is to chop the data into pieces and put each piece on its own database. You pick a shard key — say user_id — and a rule (user_id % N) that decides which database a given row lives on. Each piece is a shard, and this is horizontal partitioning.
It's powerful, and it's the most painful step in this chapter. Three reasons:
So you reach for sharding last, after a bigger box, replicas, caching, and queues have all run out of room. Here's the whole arc as a checklist — the order in which a real system peels jobs off the single box:
Section 3 claimed a stateless server is "a pure function of the request." Here's what that looks like in code. The cache and the database are the side-effecting edges — so we pass them in as dependencies instead of reaching for globals. The core cache-aside logic stays small and easy to reason about, and any number of copies can run it side by side. Flip between the two languages; the shape is identical.