Every row, message, and order needs a unique name — but a single counter handing them out is a single point of failure. This chapter builds an ID factory that runs on every machine at once, with no one in the middle.
Here's the deceptively simple ask: every time anything happens in your system — a new user signs up, an order is placed, a tweet is posted — hand back a name for it that no other event will ever share. Now do it a hundred thousand times a second, across hundreds of machines, with nobody allowed to be the central referee.
The single-server answer is a database column that counts up: 1, 2, 3, 4. Beautiful, sortable, tiny. And completely doomed the moment you have more than one machine wanting to create things at the same time — because now they're all asking the same counter, and that counter becomes the one thing every write waits on, and the one thing whose death takes the whole system down.
So we go looking for a way to mint IDs locally, on each machine, with no coordination — while still keeping them unique, compact, and roughly ordered by time. The headline trick is almost embarrassingly concrete: split a 64-bit number into chunks — some bits say when, some bits say which machine, and some bits just count within the same millisecond. That idea is called Snowflake, and most of this chapter is spent earning it.
Before reaching for any technique, pin down what we're optimizing for. The phrase "unique ID" hides at least five separate requirements, and the interesting designs trade them against each other:
BIGINT column, indexes tightly, and is half the size of a UUID. At billions of rows, that size difference is real money in storage and cache.Hold these five in mind. Each scheme below nails some and fails others — the art is knowing which failures you can live with.
The default in every relational database is an auto-increment column: the database keeps a single counter and hands out 1, 2, 3, … as rows arrive. On one machine it's perfect — unique, sortable, dense, free. The trouble starts when "one machine" stops being true.
That counter lives in one place. Every insert across your entire fleet has to go through it to get its next number, which makes it a bottleneck (everyone serializes on the same hot row) and a single point of failure (the counter's database goes down → no new IDs anywhere). You've recreated exactly the central referee we're trying to avoid.
The classic patch is multi-master with offsets: run several counters, and give each a different starting point and a shared step. Server A emits 1, 3, 5, 7…; server B emits 2, 4, 6, 8…. No collisions, and each server runs independently. It works — until it doesn't:
Auto-increment is the right answer right up until you have more than one writer. After that it fights you. We need uniqueness that doesn't require a shared counter at all.
What if each machine just rolled an enormous random number and trusted that it'll never roll the same one twice? That's a UUID (universally unique identifier): 128 bits, generated entirely locally, with no network call, no coordinator, no shared state. Any machine can mint one instantly, and the space is so vast (2122 possibilities for the random v4 variant) that the chance of two ever colliding is effectively zero.
For the "no single point of failure" and "high throughput" requirements, this is a dream — there's literally nothing to coordinate, nothing to fail, no bottleneck. But UUIDs flunk two of our five goals badly:
f47ac10b-58cc-4372-a567-0e02b2c3d479). At billions of rows that bloats every index and every cache line.The modern fix keeps the "generate locally" superpower but repairs the sorting: UUIDv7 puts a millisecond timestamp in the high bits, then random bits below. Now UUIDs sort by time and insert near the end of the index — the same core idea we're about to formalize with Snowflake, just at 128 bits instead of 64.
There's a middle path between "one central counter" and "pure local randomness": keep a central authority, but make machines talk to it rarely. A ticket server owns a single counter, but instead of handing out one ID per request, it hands out a whole range: "you get 1–1000, the next caller gets 1001–2000." Each machine then burns through its block locally, no network needed, and only comes back when it's nearly empty.
This is the deli-counter trick: you don't queue once per item, you grab a ticket for a hundred items at once. The math is the whole point — if the central server gives out blocks of 1,000, it sees only 1/1000th of the traffic, so a counter that could never survive direct write load becomes comfortable. Flickr famously used exactly this pattern. It keeps IDs compact and globally unique, and the central counter stays simple.
But be honest about what it is: you've reintroduced a single point of failure, just one that's hit far less often. If the ticket server is down when a machine exhausts its block, that machine can't create anything. The standard mitigation is to run two ticket servers (odds and evens, again with offsets) so either can fail — which brings back a little of the offset brittleness. Batching shrinks the problem; it doesn't delete it.
Here's the move that wins. Instead of one undivided number, treat a 64-bit integer as a little record with named fields, and pack three pieces of information into it. Twitter named this scheme Snowflake, and the layout is the whole idea:
Read it as a sentence: "at this millisecond, machine this produced the nth ID." That single 64-bit number is decentralized (every machine generates its own, the machine-ID bits guarantee no two nodes collide), time-sortable (the timestamp sits in the high bits, so bigger number ≈ later), and compact (it's a plain 64-bit integer). It hits four of our five requirements cleanly, and the fifth — "no single point of failure" — is satisfied because there's no central generator at all; the only shared decision is assigning each machine its 10-bit ID once at boot.
Use the builder below to feel the bit budget directly. You only have 63 usable bits to spend; give more to one field and another must shrink.
The order of the fields isn't arbitrary — it's the difference between an ID that sorts by time and one that's just random. Each field is placed to do a specific job:
And because each field is a fixed number of bits, each has a hard ceiling — rollover:
Snowflake's whole correctness rests on one assumption: each machine's clock only ever moves forward. That assumption is, in the real world, a lie — and the ways it breaks are the most dangerous failures in the entire scheme.
The mild problem is NTP drift. Machines sync their clocks over the network (NTP), and between syncs their clocks wander a little. Different machines drift in different directions, so "sortable across machines" becomes "sortable to within a few milliseconds." That's usually fine — we promised roughly time-ordered, not perfectly.
The killer is the clock going backwards. NTP can yank a fast clock back to correct it; a leap second can repeat a second; a paused VM or a laptop waking from sleep can resume in the past. When the clock jumps back, the generator starts producing timestamps it has already used — and now it can emit an ID identical to one it minted before (a duplicate), or one that sorts before an earlier ID (out-of-order). Both silently corrupt the very guarantees the ID exists to provide.
The defense is simple and strict: remember the last timestamp you emitted. If the clock ever reads earlier than that, refuse to generate — either spin-wait until the clock catches back up, or, if the jump is large, throw an error and let the node be pulled out of rotation. Never, ever mint an ID from a timestamp you've already passed. Try it in the breaker below: drag the clock backwards with protection off, and watch duplicates appear in red.
We keep saying "roughly time-ordered." The precise term is k-sorted: any ID is at most k positions away from its true time-sorted spot. Across one machine, Snowflake IDs are perfectly sorted; across machines, clock drift makes them k-sorted with a small k (a few milliseconds of slop). That "roughly" buys you a lot:
ORDER BY id DESC — no separate created_at index needed.OFFSET, no skipped or duplicated rows when new data arrives mid-scroll.The same idea travels well beyond 64 bits. ULID (Universally Unique Lexicographically Sortable Identifier) is a 128-bit cousin: 48 bits of millisecond timestamp in the high bits, 80 bits of randomness below, encoded as a 26-character string that sorts the same lexicographically as it does numerically. No machine-ID coordination needed (randomness handles collisions), it's URL-safe, and it's sortable — a nice option when you want UUID's zero-coordination and Snowflake's ordering, and don't mind 128 bits. (UUIDv7, from section 3, is the standardized version of the same recipe.)
So here's the whole landscape, as a recap of which requirements each scheme actually meets:
There's no universal winner — only the right trade for your constraints. If you control machine assignment and want the tightest 64-bit IDs, Snowflake. If you can't coordinate machine IDs and don't mind the width, ULID/UUIDv7. The thing they all share, and the thing this chapter was really about, is moving uniqueness out of a central counter and into the structure of the ID itself.
Snowflake's encode and decode are just shifts and masks — no allocation, no state, fully deterministic. encode slides each field into its slot and ORs them together; decode shifts each field back down and masks off the bits below it. This is the exact math the bit-budget widget runs: 22 = 10 + 12 (machine + sequence bits sit below the timestamp), and 12 is the sequence width. Flip between the two languages; the bit operations line up one-to-one.
encode and decode are total, pure functions on integers — same inputs, same output, no clock, no I/O. The only impure part of a real generator is reading the clock and bumping the sequence; isolate that and the dangerous code shrinks to a few lines. Purity is what makes the bit math trivially testable: decode(encode(t,m,s)) == (t,m,s).