An ad click aggregator turns an endless firehose of click events into running counts — clicks per ad per minute, top-N ads, filtered every which way — fast enough to act on. This is the canonical stream-processing problem: windowing, late data, exactly-once, and keeping a fast-but-fuzzy number honest.
A single ad on a big platform can be clicked millions of times an hour. Advertisers want to see — almost immediately — how their campaigns are doing, and the platform needs those numbers to bill correctly and to spot fraud. So the job is: take an unbounded stream of click events and keep accurate running totals, grouped by ad and by time, that anyone can query in seconds.
This sounds like a database problem, but it isn't really. The clicks never stop arriving, so you can't just "run a query at the end" — there is no end. Instead you process the stream as it flows, folding each event into a running tally. That shift — from querying data at rest to continuously aggregating data in motion — is what stream processing is, and this chapter is the canonical example of it. It builds directly on the message queue from Chapter 9: the queue is the firehose; everything here is what you bolt onto the end of it.
Before any boxes-and-arrows, pin down what the system must answer. For an ad click aggregator the core queries are narrow and well-defined:
ad_id and a minute, how many clicks? This is the atomic unit everything else is built from.Then the question that shapes the whole architecture: how fresh, and how exact? These two pull against each other.
The standard answer is to provide both: a fast approximate number for the dashboard, and a slower exact number for billing that corrects the fast one later. Hold that thought — it's the Lambda architecture in section 5.
The shape of every system like this is a conveyor belt of stages, each doing one job and handing off to the next. Reading left to right:
{ad_id, ts, user, country, device}.(ad_id, minute). This is the heart of the system (sections 3–4).One more box, easy to forget and impossible to live without: keep the raw events too. Alongside the aggregation, write every raw click to cheap bulk storage (a data lake on blob storage or HDFS). You'll need it when — not if — you have to reprocess: a bug in the aggregation logic, a new attribute you forgot to count, or a billing dispute. Aggregates are derived; raw events are the source of truth. Throw away the source and a wrong number can never be made right.
"Clicks per minute" implies cutting the never-ending stream into minute-sized buckets and counting each. That cut is a window, and there are two common kinds:
Now the hard part. Events arrive over a network from millions of devices, so they show up out of order and late. A click that happened at 12:00:59 might not reach your aggregator until 12:01:30 — after you'd want to close and report the 12:00–12:01 window. When do you decide a window is "done"?
The answer is a watermark: a moving threshold that says "I believe I've now seen all events up to time T." When the watermark passes a window's end, you close that window and emit its count. The watermark trails real time by a grace period (say, 30 seconds) to wait for stragglers. Set it too tight and you drop late clicks; too loose and your numbers lag. Mechanically, aggregation is map-reduce in miniature: map each event to a key (ad_id, ts/60_000), then reduce by summing — partial sums computed per partition, combined into the final count.
A queue like Kafka gives you at-least-once delivery: it guarantees a message won't be lost, but it may be delivered more than once. If a consumer reads a batch, adds it to the counts, then crashes before recording how far it got, it'll re-read that batch on restart and count those clicks twice. For billing, double-counting is as bad as losing.
Turning at-least-once into effectively-once takes three cooperating tricks:
"Exactly-once" is really "at-least-once delivery + idempotent, checkpointed processing." Nobody delivers exactly once over an unreliable network; you achieve the effect of exactly-once by making redundant deliveries harmless.
f(state, event) where f(f(s, e), e) == f(s, e) means you can safely replay the tail of the stream after any crash. Purity + idempotence is what makes "just replay it" a valid recovery strategy instead of a corruption risk.Back to the freshness-vs-exactness tension. The classic resolution is the Lambda architecture: run two paths over the same event stream.
The serving layer merges them: show the speed layer's number now, then overwrite it with the batch layer's exact number once it's ready. The fast number gets quietly corrected — this reconciliation is the point of the whole design.
The cost of Lambda is real: you maintain two codebases that must compute the same thing the same way, or the corrected number contradicts the live one. The Kappa architecture answers this by deleting the batch layer entirely: keep one stream-processing codebase, and when you need to "recompute," just replay the stream from the start (the queue is retained and replayable) through the same code. Correction becomes "rewind and re-run," not "maintain a second system." Kappa is simpler to operate; Lambda can be cheaper for genuinely huge historical recomputes. Both rely on the same foundation: a durable, replayable log of raw events.
The pipeline ends in two very different storage systems, because aggregates and raw events have opposite needs.
Why both? The aggregation store is fast but derived and lossy — it's just sums, you can't change what you counted after the fact. The raw store is slow but complete and authoritative — it's the receipts. Keep the raw store and any aggregate can be rebuilt or audited; lose it and a wrong total is wrong forever. It's also your defense in a billing dispute: "here is every click we charged you for."
To handle the firehose, you split work across many aggregator instances by partitioning the stream by ad_id: all clicks for a given ad go to the same partition (and the same aggregator), so its running count lives in one place and never needs cross-machine coordination. Add partitions, add throughput. Clean — until the load isn't even.
Real ad traffic is brutally skewed. One viral ad or one Super Bowl spot can pull 90% of all clicks. Partition by ad_id and every one of those clicks lands on a single partition, which melts while the others idle. This is the hot-key / celebrity problem from sharding (Ch.1), back again in stream form.
The fix is key salting: split the hot key artificially. Instead of ad_id, partition by (ad_id, salt) where salt is a small random number 0–9. Now the hot ad's clicks spread across ten partitions, each keeping a partial count; a cheap second step sums the ten partials back into the true total. Load spreads; the answer stays correct. Two more must-haves at scale:
A click aggregator's numbers go straight into invoices, so "roughly right" isn't good enough at billing time. Four habits keep it honest:
Here's the whole chapter as the order you'd actually build it:
(ad_id, minute) counts; bucket by event time.Strip away the queues and machines and the core of this whole chapter is four lines: bucket each click by (ad_id, ts / windowMs), then count by key. That's a fold into a map keyed by (entity, window) — and it's the same fold whether it runs in the real-time speed layer or the from-scratch batch layer, which is exactly why a batch recompute and a live stream can agree. Flip between the languages; the shape is identical.
ts / windowMs — is the tumbling window; the (+) combine is associative, so partial counts from different partitions merge cleanly (that's what makes salting and map-reduce work). Same pure function in the speed layer and the batch layer means the corrected number can never disagree in kind with the live one — only be more complete.