A distributed message queue is, at heart, one stubbornly simple idea: writers append messages to the end of a never-ending file, and readers each remember how far they've read. This chapter builds a mini-Kafka from that single primitive — partitions, replication, consumer groups, and the delivery guarantees that fall out of it.
Every chapter before this one peeled a job off the single box. This one zooms into the humble queue we added in Chapter 1 and asks: what if that buffer became the system's backbone — durable, replayable, and able to feed a hundred independent readers at once?
That system exists, and you've used it without knowing: Kafka, Pulsar, Kinesis, and their kin move the world's event streams. They look intimidating from the outside — partitions, ISRs, KRaft, exactly-once. But underneath there is one data structure doing almost all the work: an append-only log. A file you only ever add to the end of, and that you read by remembering a position.
We'll build the whole thing up from that primitive. First why a queue earns its place at all, then the core abstractions, then how the log lives on disk, how it survives a machine dying, how many readers share it, what "delivered" really means, and finally who keeps the cluster honest. Plain English first; the Kafka vocabulary arrives only once the idea behind it already makes sense.
Imagine your web app, the moment a user clicks "post." It needs to: save the post, update twelve followers' timelines, generate a thumbnail, run a spam check, and fire an analytics event. Do all of that inline and the user stares at a spinner for two seconds. Worse, if the analytics service is down, the whole click fails — even though analytics had nothing to do with whether the post saved.
The fix is to decouple. The web app does the one essential thing (save the post) and then drops a little "this happened" message onto a shared list. Other services read from that list and do their slow or optional work whenever they're ready. The writer doesn't know or care who reads, or when, or how many readers there are. That shared list is the queue, and three big wins fall out of it:
That last point is the crucial distinction. A traditional message broker (think old-school RabbitMQ semantics) treats a message like a letter: once a consumer takes it, it's gone. A log-based queue like Kafka treats messages like entries in a ledger: reading one doesn't erase it. A reader just moves its bookmark forward. Many independent readers can hold many different bookmarks in the same log at the same time.
Three nouns carry the whole model. Learn them and the rest is detail.
"orders", "clicks", "signups". It's the logical channel producers write to and consumers read from.The cast of characters: a producer sends messages to a topic; a broker is a server that hosts partitions and stores their logs on disk; a consumer reads messages starting from an offset. Producer → broker → consumer, with the broker's append-only log as the thing in the middle that everyone agrees on.
The genius is what's missing. The broker keeps no per-consumer state about what's been delivered or acknowledged — it just stores bytes in order. All the "where have I read to" lives on the consumer side as an offset. That asymmetry is why one Kafka cluster can fan one topic out to hundreds of consumers cheaply: the broker isn't tracking any of them.
alice always lands in the same partition, because the partition is chosen by hash(key) % partitionCount. That determinism is what gives you per-key ordering — the topic of the next section.Here is the single most misunderstood fact about Kafka, so we'll say it plainly: order is guaranteed only within one partition, never across the whole topic. Inside partition 2 the messages are 0, 1, 2, 3 in iron sequence. But message 5 in partition 0 and message 5 in partition 1 have no defined order relative to each other — they were written in parallel by different machines.
So how do you make sure related messages stay in order? You give them the same partition key. Every message can carry a key, and the broker routes it by hash(key) % partitionCount. Use user_id as the key and every event for a given user flows through one partition, in order. Different users may scatter across partitions, but each user's own timeline is perfectly sequenced. You get ordering exactly where you need it and parallelism everywhere else.
And parallelism is the whole point of partitions. The number of partitions is your unit of parallelism: a topic with 12 partitions can be drained by up to 12 consumers at once, each owning some partitions. Too few partitions and you can't add readers to go faster; too many and you pay overhead. Partition count is one of the few Kafka decisions that's genuinely hard to change later, so it's worth getting roughly right up front.
"Writing to disk is slow" is true for random writes — seeking the disk head all over the place. But a log only ever appends to the end, which is a sequential write, and sequential disk throughput is enormous (often faster than random RAM access, once you count it in megabytes per second). Choosing append-only isn't just for replay — it's what makes the log fast in the first place.
A partition's log isn't one giant file; it's a series of segment files of a fixed size. New messages go into the active segment; when it fills, the broker rolls to a fresh one. This makes deletion trivial: to enforce retention (keep the last 7 days, or the last 50 GB), you just drop whole old segment files — no expensive in-place edits.
Two more tricks make reads cheap:
sendfile), skipping the usual copies through application memory. Less CPU, less garbage, more throughput.Finally, log compaction is an alternative to time/size retention for "latest value" topics. Instead of deleting old messages by age, the broker keeps only the most recent message for each key and garbage-collects the earlier ones. A compacted topic becomes a durable, replayable snapshot of current state — perfect for "the current address of every user" rather than "every address change ever."
A log on one disk dies with that disk. So each partition is copied onto several brokers. One copy is the leader; the rest are followers. All reads and writes for a partition go to its leader; the followers do nothing but continuously pull the leader's new messages and append them to their own copies, racing to stay caught up.
The followers that are caught up (within a small allowed gap) form the in-sync replica set, the ISR. The ISR is the live roster of copies that are trustworthy enough to take over. If a follower falls too far behind or its broker hangs, it's kicked out of the ISR; when it catches up again, it rejoins. The ISR shrinks under stress and grows back as things recover.
Now the durability dial: when a producer sends a message, when do we tell it "got it"? That's the acks setting:
acks=0 — fire and forget. The producer doesn't wait at all. Fastest, but a message can vanish if the leader dies before storing it.acks=1 — wait until the leader has written it. Safe against most failures, but if the leader dies before any follower copied that message, it's lost.acks=all — wait until every in-sync replica has the message. Now the message survives any single broker (indeed any failure short of losing the whole ISR). Most durable, highest latency, because you wait for the slowest replica in the ISR.To keep consumers from ever reading a message that might still be lost, the broker exposes only up to the high-watermark: the highest offset that all ISR members have. Messages above the watermark exist on the leader but aren't yet visible to consumers. When the leader dies, a follower from the ISR is elected the new leader — and because every ISR member is at least at the high-watermark, no acknowledged, visible message is lost in the handover.
acks=all only guarantees as much as the ISR is wide. If the ISR has shrunk to just the leader, "all" means "one." That's why production clusters also set min.insync.replicas ≥ 2 — refuse the write rather than acknowledge something that isn't actually replicated.One consumer might not keep up with a busy topic. So consumers band into a consumer group: a team that shares the work of one topic by dividing its partitions among themselves. The rule is clean — each partition is assigned to exactly one consumer in the group at a time. Three consumers, twelve partitions: each gets four. The group as a whole reads every message once; no two members process the same partition.
The group's progress is stored as a committed offset per partition (Kafka keeps these in a special internal topic). When a consumer finishes processing up to offset 4,000 on its partition, it commits that number. If it crashes and another consumer takes over, the replacement resumes from the last committed offset — that's how the group remembers its place across restarts.
What happens when a consumer joins or leaves (a deploy, a crash, a scale-up)? The group runs a rebalance: partitions are revoked from everyone and re-divided so each surviving member gets a fair share. A group coordinator (a broker) orchestrates it; members briefly pause, get their new assignments, and resume. Rebalances are necessary but disruptive — during one, processing stalls — so newer protocols (cooperative/incremental rebalancing) try to move only the partitions that must move instead of stopping the world.
And the symmetry to remember: separate groups are fully independent. Group "billing" and group "analytics" each read the whole topic from their own offsets, oblivious to each other. That's the multi-reader replay power from Section 1, in operational form.
The hardest honesty in distributed systems: a message and an acknowledgement can both get lost, and the sender can't tell which happened. From that uncertainty come three possible guarantees, and you pick one by choosing when the consumer commits its offset.
That's the deep idea worth carrying away: "exactly-once" is really "atomic append + offset-commit." If advancing your cursor and recording your output happen as one indivisible transaction, then a crash can never leave you having done one without the other — so there's nothing to duplicate and nothing to lose.
One more design choice underlies all of this: Kafka consumers pull. Rather than the broker pushing messages at consumers (and risking overwhelming a slow one), each consumer asks for the next batch when it's ready. Pull means the consumer sets its own pace — natural backpressure. The price is consumer lag: the gap between the log's tail and a consumer's offset. Watching lag is how you know whether your consumers are keeping up; growing lag means add capacity or the stream will outrun you.
(offset, sideEffects). At-least-once advances the offset after the effect; at-most-once before. Exactly-once makes the offset update and the effect a single atomic transaction — the functional move of "commit both or neither," lifted onto a stream.A cluster of brokers needs to agree on facts that change as machines come and go: which brokers are alive, which broker is the leader for each partition, what topics and partitions exist, and where a producer should send a given write. Someone has to be the source of truth for this cluster metadata, and crucially, everyone must agree on it even as nodes fail.
That's a consensus problem (Chapter 6 territory). Historically Kafka outsourced it to ZooKeeper — a separate, strongly consistent coordination service that stored membership and leadership and ran leader elections. It worked, but it meant operating a second distributed system alongside your first, with its own failure modes.
Modern Kafka folds coordination inward with KRaft (Kafka Raft). A small set of brokers act as controllers and run the Raft consensus protocol among themselves to maintain the cluster metadata as its own internal log — the same append-only primitive this whole chapter is built on, now storing "who leads partition 7" instead of user events. One controller is the active leader; it decides partition leadership and broadcasts the metadata to every broker. Fewer moving parts, faster failover, and a pleasing self-similarity: the log coordinates the logs.
orders is led by broker B5") from any broker, then talks straight to the leader. When leadership moves, the producer's next metadata refresh redirects it — clients re-discover the topology rather than hard-coding it.The architecture is elegant; keeping it healthy is where the work lives. A few realities operators face:
hash(key) % count, so existing keys may jump partitions and per-key ordering can break across the boundary. Plan partition count generously up front.acks=all durability is quietly evaporating. Alert on both.Here's the whole chapter as a checklist — the path from "a list you append to" to a production event backbone:
The whole chapter compresses into a few lines. A log is an immutable sequence you only append to. A consumer is a cursor — literally one integer offset — and reading is poll: hand back the message at the current offset (if any) and a cursor advanced by one. Nothing about the log knows the cursor exists; that's why a hundred cursors can read it independently. Flip the languages; the shape is identical.
append (or side-effect) and the offset-commit atomic and idempotent, so a crash can never split them. Append once, advance once — or neither.