System Design consistency & CAP
⚖️ Part I · Foundations · chapter 4 / 24

When the network breaks,
what do you give up?

When two of your servers can't talk to each other, you have to choose: keep answering with maybe-stale data, or refuse to answer until they reconnect. This chapter turns that one painful choice into a vocabulary — CAP, PACELC, and the spectrum of consistency models in between.

Every distributed system makes one promise it can't fully keep: that all its copies of your data agree, instantly, all the time. The network won't allow it. This chapter is about being honest — and deliberate — about exactly which part of that promise you break, and when.

We'll start by pinning down what "consistency" even means here (it's a slippery word with two different jobs). Then we'll see why making copies of data is what creates the whole problem in the first place. From there: CAP made concrete, PACELC's everyday correction to it, the full spectrum of consistency models, the session guarantees that fix the most common bug, and finally how real systems tune the knob — strong for money, eventual for feeds.

1What "consistency" even means here

"Consistency" is one word doing two unrelated jobs, and conflating them causes endless confusion. Before we go further, let's separate them.

  • The database / ACID sense. Here "consistent" means a single database never breaks its own rules — a transaction moves it from one valid state to another, foreign keys hold, constraints are honored. This is about internal integrity inside one node. It's the C in ACID.
  • The distributed-systems sense. Here "consistent" means do all the copies agree on the latest value? You wrote balance = 50 on one machine; if I read from another machine right now, do I see 50, or do I see the old 40? This is the C in CAP, and it's the one this whole chapter is about.

Plain-English version of the distributed sense: "if I just changed something, will everyone — including me — see the change?" When the answer is "always, immediately," you have strong consistency. When the answer is "eventually, give it a moment," you have eventual consistency. Everything in this chapter lives between those two poles.

⚠️
Don't mix the two C's. A database can be perfectly ACID-consistent on each node and still be wildly inconsistent across nodes — every replica is internally valid, they just disagree about the latest value. When someone says "is it consistent?", always ask: which consistency?

2Replication creates the problem

Here is the uncomfortable truth: if your data lived in exactly one place, on one machine, there would be no consistency problem at all. There's only one copy, so it can't disagree with itself. The instant you make a second copy, you've created the possibility that the two copies hold different values — and the whole tension of this chapter is born.

But you can't keep just one copy. Chapter 1 already forced your hand: you replicate the database for read capacity, and you spread copies across regions so a single data-center fire doesn't end the company. Replication isn't optional — it's the price of being fast and surviving failure. And the moment data lives in more than one place, copies can disagree.

Why do they disagree? A write lands on one copy and has to travel to the others, and that trip takes time — milliseconds across a rack, tens of milliseconds across a continent. During that travel window, a reader hitting a not-yet-updated copy sees stale data. Network hiccups, slow links, and outright outages stretch that window from "barely noticeable" to "indefinite." Consistency, then, is fundamentally a question about how you manage the gap between copies — how long you tolerate it, and what you do when the gap can't close.

λ
The FP framing: one immutable value can never be inconsistent with itself — consistency problems are replication problems, full stop. Think of each replica as a separate copy of a value that's being independently updated; "consistency" is just asking how and when those copies are reconciled back into agreement.

3CAP, made concrete

The CAP theorem names three things you'd love to have, all at once:

  • C — Consistency. Every read sees the most recent write, no matter which copy answers it. (This is the distributed sense from §1.)
  • A — Availability. Every request gets a non-error answer, even if that answer might be slightly old. The system never says "go away."
  • P — Partition tolerance. The system keeps working even when the network drops messages between nodes — when, say, the link between two data centers goes down and your copies temporarily can't talk to each other. That split is a partition.

The pop-culture version — "pick any two of three" — is misleading. Here's the honest version. Partitions are not optional. Networks really do drop packets, cables really do get cut, and a switch really does reboot mid-deploy. You don't get to "choose" not to tolerate partitions any more than you get to choose not to have weather. So P is a given. The real theorem is about what happens during a partition:

When a partition splits your nodes, you can keep one of C or A, not both. Either a node on the cut-off side keeps answering with the data it has (choosing A, possibly stale), or it refuses to answer until it can confirm it's current (choosing C, becoming unavailable). That's the entire choice.

Concretely:

  • CP — pick consistency. A bank balance, an inventory count, a "did this payment go through?" check. Better to return an error or stall than to hand someone a wrong number. During a partition the minority side stops serving.
  • AP — pick availability. A "like" counter, a view count, a social feed, a shopping cart. A slightly-stale like count is fine; a 500 error is not. During a partition every side keeps answering and the copies reconcile later.
⚠️
Myth-busting: you never "trade away P." In any system that spans a network, partitions will happen whether you planned for them or not. A so-called "CA" system just means a single-node system, or one that corrupts/hangs when the network splits. For anything distributed, the live question is only ever CP vs AP.
Interactive · partition simulator (pick C or A) Cut the link, write to one side, then choose how the other side behaves
link up · both sides agree

4PACELC — the part CAP forgets

CAP only talks about the dramatic moments — partitions — which, for a well-run system, are rare. But there's a tradeoff you pay all day, every day, even when the network is perfectly healthy, and CAP says nothing about it. PACELC fixes that omission.

Read it as two clauses:

  • PAC — "Partition? then A or C." (This is just CAP: during a partition, choose availability or consistency.)
  • ELC — "Else (no partition), L or C." Even with the network fully healthy, you still choose between low Latency and strong Consistency.

Why is there still a choice when nothing is broken? Because making copies agree before you answer takes time. If you want a read to reflect the very latest write, you may have to wait for several replicas to confirm they're current — that's extra round trips, that's latency. If instead you answer from the nearest replica right now, you're fast but you might serve data that's a few milliseconds behind. Strong consistency literally costs milliseconds on every single request.

That's why PACELC matters more day-to-day than CAP: partitions are occasional, but the latency-vs-consistency dial is turned on every read and write you ever serve. The notation pairs the two choices, e.g. PA/EL ("available under partition, low-latency otherwise") or PC/EC ("consistent always, latency be damned").

λ
The FP framing: consistency is a synchronization barrier — a point where parallel copies must agree before proceeding. Barriers are correct but they cost wall-clock time, because the fast workers wait for the slow ones. Latency-vs-consistency is just how loose you let that barrier be.
Interactive · PACELC decision tree Walk the branches; land on a real-world system label
Start: is there a partition right now?

5The spectrum of consistency models

"Strong vs eventual" is a useful headline, but the real world is a gradient with many named stops in between. Each stop is a different promise the system makes about what your reads can see. From strongest (and slowest) to weakest (and fastest):

  • Everyone sees every change the instant it happens, as if there were one global clock.Linearizable / strong consistency. The gold standard: every operation appears to take effect at a single point in time, and reads always see the latest write. Most expensive.
  • Everyone agrees on the order of operations, even if not on exactly when.Sequential consistency. All nodes see the same sequence of writes; that sequence just might lag real time.
  • If event B was caused by event A, nobody sees B without first seeing A.Causal consistency. Cause comes before effect everywhere (your reply never shows up before the comment it answers), but unrelated events can be seen in any order.
  • You always see your own latest changes.Read-your-writes. Others might lag, but you never lose track of what you just did. (The fix for the §6 bug.)
  • Once you've seen a value, you never see an older one.Monotonic reads. No "time travel" backward — refreshing never shows you data older than what you already saw.
  • Wait long enough with no new writes, and all copies converge.Eventual consistency. The weakest useful promise: agreement eventually, with no bound on when. Cheapest and most available.

Read-your-writes and monotonic-reads belong to a friendly family called session guarantees — promises scoped to one user's session rather than the whole system. They're cheap to provide and they kill the most-felt anomalies, which is why they're the next section.

6Read-your-writes & session guarantees

Here's the anomaly everyone has felt: you post a comment, the page refreshes, and your comment is gone. You didn't do anything wrong. What happened is the write went to the primary, but your refresh's read got routed to a replica that hadn't caught up yet — so it honestly reported "no comment here." A few seconds later it appears, which makes it feel like a ghost.

The full strong-consistency fix (make every reader everywhere see your write instantly) is expensive overkill. You don't need the whole world to see your comment immediately — you just need you to. That narrower promise is read-your-writes consistency: within your own session, any read reflects your own prior writes.

It's cheap to implement because it's scoped to one session. Common tricks:

  • Read from the primary for a moment — right after a user writes, route their reads to the primary (which is always current) for a few seconds, then let them drift back to replicas.
  • Sticky reads — pin a user's reads to the same replica they wrote through, so they at least never go backward.
  • Version tokens — hand the client a "you've seen up to version 1234" token; reads carry it, and a replica that's behind 1234 either waits or forwards the request.

Pair read-your-writes with monotonic reads (never show older data than already seen) and the everyday experience feels rock-solid — even though, behind the curtain, the system is still only eventually consistent for everyone else.

Interactive · consistency-model timeline Two clients, one key — see which reads are fresh vs stale per model
A writes v2 · then both clients read

7Tuning the knob in practice

Consistency isn't a single switch you flip for the whole system — it's a dial, and modern stores let you turn it per operation. The mechanism most systems expose is the quorum.

Say each piece of data is stored on N replicas. On a write you require W of them to acknowledge before you call it done; on a read you ask R of them and take the newest answer. The magic rule (we'll prove it in Chapter 7) is:

W + R > N ⟹ strong consistency

If the write set and the read set must overlap, then any read is guaranteed to touch at least one replica that saw the latest write — so it can't miss it. Crank W and R up and you get strong consistency at the cost of latency; drop them (e.g. W=1, R=1) and you get blazing speed but only eventual consistency. Same cluster, different dial position per request.

That's why the right mental model isn't "is my system CP or AP?" but "which consistency does this particular operation need?" Almost every real product mixes both:

  • Strong, for money and identity. Account balances, payments, inventory decrements, "is this username taken?" — anything where a wrong answer costs real money or breaks an invariant. Pay the latency.
  • Eventual, for feeds and counters. Likes, view counts, timelines, recommendations, "who's online" — high-volume, low-stakes data where a brief lag is invisible and availability is king.

The same company runs both at once: Amazon keeps your shopping cart highly available (AP — never lose an add-to-cart), but processes the final payment with strong consistency (CP — never double-charge).

The code below models the read side of a quorum: given the values returned by the replicas you queried, decide what to hand back. Two strategies, both common in the wild — last-write-wins (trust the highest timestamp) and a max-merge (for a value that only ever grows, like a counter, take the largest). Flip languages; the shape is identical.

λ
The FP framing: reconciling replicas is just a fold with an associative, commutative merge. If your merge is a join on a lattice (last-write-wins, or "take the max"), order doesn't matter and replays are harmless — that's the math behind CRDTs, the data types that converge without coordination.

8Picking a point on the spectrum

Putting it all together, here's a quick decision guide. Walk it per operation, not per system.

  1. 1Does a wrong/stale answer cost money or break an invariant? (balance, payment, inventory, unique username) → strong / CP. Pay the latency; use a high quorum (W+R>N) or a single-leader store.
  2. 2Is it high-volume, low-stakes, and must never be down? (likes, views, feeds, presence) → eventual / AP. Favor availability and low latency; reconcile with a commutative merge.
  3. 3Does the user just need to see their own changes? (post a comment, edit a profile) → read-your-writes / session consistency. Cheap; route their reads to the primary briefly or pin to one replica.
  4. 4Must cause precede effect across users? (chat replies, threaded comments) → causal consistency. Track happens-before; don't show effects before their causes.
  5. 5Is occasional time-travel-backward unacceptable? (a refresh showing older data) → add monotonic reads on top of whatever else you chose.

The senior move isn't memorizing CAP — it's noticing that "consistency" is a per-operation budget, and spending it where wrong answers hurt while saving it everywhere else.