A web crawler is one tiny loop — pull a URL, fetch the page, find its links, add them back — run a billion times across thousands of machines. The whole art is in the bookkeeping: not crawling the same page twice, not hammering any one site, and not falling into traps that never end.
Strip a web crawler down to its bones and you get a loop a first-year could write: take a URL off a list, download the page, pull out the links, put the new ones back on the list, repeat. The reason "design a web crawler" is a real interview question — and the reason Google's runs on warehouses of machines — is everything that loop has to avoid.
It must not download the same page a million times. It must not knock a small blog offline by requesting every page at once. It must obey each site's robots.txt rules. And it must not wander into a calendar that generates "next month" links forever, or a search box that spits out infinite URLs. None of that is hard math; all of it is careful bookkeeping at a scale where careless bookkeeping melts the internet.
This chapter builds the crawler the way the rest of the book builds systems: start with the one honest loop, then add exactly the machinery each new problem demands — a structured queue for politeness, two dedup layers, a robots cache, trap defenses, and finally the sharding that lets thousands of crawlers cooperate without tripping over each other.
Everything starts with a handful of seed URLs — the doors you walk through first, like a few big directories or your own homepage. From there the loop turns, and each turn passes a URL through the same relay of small parts:
/about → https://site.com/about).That's the canonical web-crawler diagram, and it is genuinely the whole thing. Notice the two independent "have we seen this?" gates — one on the URL, one on the content. Keeping those straight is half of §5. The other structural fact: the loop is a closed circle. Links found while crawling become the seeds of the next crawl, which is why a crawler with good seeds eventually reaches almost everything reachable.
frontier (a queue) and seen (a set). Each step dequeues one URL, fetches, and folds the new links back in. The messy part — the actual HTTP, the DNS — is pushed to the edges as effects; the core is a pure step. We write exactly that in §code.If the frontier is just a plain queue — first in, first out — the crawler naturally does breadth-first search: it crawls all the seeds, then everything one link away, then everything two links away, spreading outward in rings. Swap the queue for a stack (last in, first out) and you'd get depth-first search: follow one chain of links as deep as it goes before backing up. Crawlers almost always pick BFS, because depth-first tends to burrow into one site forever, and breadth-first naturally spreads attention across many sites.
But a plain FIFO queue has a fatal flaw the moment you care about manners. Pages on the same site link heavily to each other, so a simple BFS pulls a long run of URLs all from one host — and fires them off back-to-back, which is exactly the hammering we must avoid. Plain BFS is also blind to importance: the homepage of a major news site and a random forum reply sit in the queue as equals.
So real crawlers replace the single queue with a structured URL frontier built from two banks of queues:
blog.com lands in the same queue. A worker pulls from a back queue and waits between requests, so one host is never hit too fast (§3).The frontier, then, is the crawler's brain. It answers one question — "which URL next?" — while honoring two constraints at once: crawl the important things first, and never be rude to any single host.
A crawler can open thousands of connections per second. A hobbyist's blog runs on one small server. Point the first at the second without restraint and you've effectively launched a denial-of-service attack — which is rude, often illegal, and a fast way to get your crawler banned everywhere. Politeness is the set of self-imposed limits that keep a crawler a guest rather than a wrecking ball.
Two rules do most of the work:
blog.com, wait a beat — a second or two, or whatever the site's robots.txt requests — before fetching the next page from that same host. Meanwhile you're free to fetch from a thousand other hosts in parallel.That last point is the key insight: politeness is per-host, not global. You stay busy by spreading across many hosts, while being gentle with each one. The frontier's back-queue design exists precisely to enforce this. Each host maps to exactly one back queue (a host → back-queue mapping), and each back queue is handed to exactly one worker thread (a queue → worker assignment). Because a host's URLs all funnel through one queue served by one worker that sleeps for the crawl-delay between pulls, the "one connection, polite spacing" guarantee falls out automatically — no global locking required.
The web is far too big to crawl evenly, so the front queues exist to crawl the good stuff first. A prioritizer scores each URL and the score decides which front-queue tier it lands in. Common signals:
That last signal drives the crawler's second job: not just discovering pages once, but keeping them current. This is freshness, and it's a scheduling problem. You don't recrawl everything on a fixed timer — that wastes most of your fetches on pages that haven't moved. Instead you track each page's historical update cadence: how often did it actually change last time you looked? A page that changes every hour gets recrawled hourly; one that's been identical for two years gets checked rarely. Recrawl scheduling tunes itself toward where change actually happens, so a fixed crawl budget captures the most new information.
The web is enormously redundant. The same URL is linked from a thousand pages; the same article appears on a dozen mirror sites; the same page is served with and without a trailing slash, with and without tracking parameters. A crawler that doesn't dedup does the same work over and over and fills its storage with copies. So it dedups at two independent levels.
Before adding a URL to the frontier, check whether we've already seen it. First normalize the URL so trivially-different spellings collapse to one: lowercase the host, drop the default port, sort query parameters, strip tracking junk, resolve /a/../b to /b. Then look it up in a seen-set.
At web scale that set has hundreds of billions of entries — too many to hold every full URL in memory. So crawlers reach for a Bloom filter: a compact bit array that answers "have I seen this?" using a fraction of the memory. The trade is a strange one — it can have false positives (claim you've seen a URL you haven't, so you skip a genuinely new page) but never false negatives (it never forgets something it really saw). For a crawler that's an acceptable bargain: occasionally missing a page is fine; the memory savings are enormous. Play with one below.
Two different URLs can serve byte-for-byte identical pages, and near-identical pages (same article, different ad) are everywhere. After fetching, compute a fingerprint of the content and check whether you've stored that fingerprint already. A plain hash catches exact duplicates. For near-duplicates you use SimHash: a fingerprint engineered so that similar documents get similar fingerprints, so "almost the same page" collides and gets dropped. URL-seen stops you from re-fetching; content-seen stops you from re-storing.
Every well-behaved crawler honors an unwritten contract with the sites it visits, and the contract is written in a file. Before crawling any host, fetch https://host/robots.txt — a small text file where the site declares which paths crawlers may and may not touch. Respecting it isn't optional courtesy; ignoring it gets you blocked and can get you sued.
Three pieces matter:
robots.txt for every page — that would itself be impolite. Fetch it once per host, cache it with a TTL, and check every candidate URL against the cached rules before queueing it.robots.txt often points to a sitemap — a list the site publishes of its own URLs. It's a gift: a curated set of seeds straight from the source, far better than guessing.The practical shape: robots.txt is one more cache keyed by host, consulted in the URL filter step. New host? Fetch its rules first, then crawl within them.
Some pages generate links forever. A calendar widget has a "next month" link, and next month has a next month, and a naive crawler will happily request the year 9999 and keep going. A faceted shop produces a fresh URL for every combination of filters — color × size × sort × page — an effectively infinite space of near-identical pages. A search box turns any query into a URL. None of these are malicious; they're just spider traps: structures that emit unbounded streams of low-value URLs and can swallow a crawler whole.
You can't detect "infinite" directly, so you defend with cheap, blunt limits:
A → B → A chains, or a page that always redirects to a slightly different URL, are traps too; cap the redirect chain.These same limits double as a quality filter: capping depth and length, and dropping near-duplicates (§5), keeps the crawler spending its finite budget on real, distinct content instead of machine-generated noise. Try escaping a trap below.
One machine crawls maybe a few hundred pages a second. The web has hundreds of billions of pages. So a real crawler is a fleet — many crawler servers running the same loop in parallel — and the design problems are all about coordination.
Here's the whole crawler as a checklist — the order in which each problem forces its piece of machinery into the design:
Section 1 claimed the loop is "a fold over a growing work-list." Here it is in code. The state is exactly two values: a frontier (queue of URLs) and a seen set. The step function is pure — it dequeues one URL, and given the links that page contained, returns the next frontier, the next seen-set, and the URL it fetched. The actual HTTP fetch (the only effect) is passed in / lives at the edge, never inside step. Flip between languages; the shape is identical.
step is a pure transition on the pair. All the messy real-world parts — HTTP, DNS, robots.txt, persistence — sit at the edges as effects feeding links into the pure core. That separation is what lets you test the crawl logic without touching the network, and run a thousand copies of the loop in parallel.