Something happens — an order ships, a friend comments — and one tiny fact has to reach a phone, an inbox, and a text message, reliably, without spamming anyone. This chapter builds the machinery that fans one event out across push, SMS, and email and gets it delivered exactly enough times.
A notification system looks deceptively easy: "when X happens, tell the user." The hard part isn't sending one message — it's sending millions of them, across channels you don't control, reliably, without ever sending the same thing twice or burying someone in noise.
Every channel is a third party with its own rules and its own bad days. Apple's push service might be fast and Twilio's SMS slow; SendGrid might shrug off a spike while a carrier rejects half your texts. Your job is to absorb all that variability behind one clean idea — "deliver this notification to this person" — and make it true.
We'll build it the way the rest of the book builds things: one concern at a time. Channels and providers, then the fanout pipeline that splits an event across them, then everything that keeps it honest — retries, dedup, rate limits, templates, and the user's right to be left alone.
A channel is a way to reach a person. There are four big ones, and you almost never talk to the device directly — you talk to a third party whose whole business is delivering on that channel.
Two things make this work. First, an integration layer per provider — a thin adapter that turns your internal "send this" into APNs' or Twilio's exact API shape, holds the credentials, and parses their responses. Second, a contact-info store: the database that maps a user to how to reach them — their device tokens (one per installed app), phone number, and email address. A user can have several device tokens at once (phone + tablet) and they go stale constantly (app reinstalled, number changed), so keeping this store fresh is a real job, not a footnote.
Here's the shape of the whole system. A business service — orders, social, billing — decides something is worth a notification and calls the notification servers. Those servers do the thinking: look up the user's contacts and preferences, figure out which channels apply, and then fan the event out — drop one message onto a separate queue per channel. Behind each queue sits a pool of workers that pull messages off and call that channel's provider.
The crucial design choice is one queue per channel, not one big shared queue. Why split them? Because the channels have wildly different personalities. APNs can absorb a huge burst in milliseconds; an SMS carrier might accept a few hundred per second and occasionally just stop. If push and SMS shared a queue, the slow carrier's backlog would clog the fast push traffic behind it — head-of-line blocking. Per-channel queues decouple the producer from the providers: the notification servers enqueue at full speed and move on, while each channel's workers drain at whatever pace that provider can handle. A backed-up SMS queue never slows down email or push.
This is the queue-and-worker pattern from Chapter 1, applied per channel. The notification servers are producers; they stay fast no matter how slow Twilio is having a bad afternoon. Spikes get absorbed by the queues; you scale a channel by adding workers to its pool only.
flatMap over channels. event.flatMap(e => channelsFor(e.user)) turns one event into a list of per-channel messages. The decision is pure; the enqueue is the only effect. Keep the split pure and you can unit-test "who gets what" without a single network call.Providers fail. Not catastrophically — just sometimes. A request times out, APNs returns a transient 503, a carrier hiccups. If you treated every failure as final you'd silently drop notifications all day. So the contract is at-least-once delivery: keep trying until the provider confirms it took the message.
But you can't retry instantly and forever — a tight retry loop against a struggling provider just makes things worse (a self-inflicted DDoS). The fix is exponential backoff: wait a little after the first failure, then twice as long, then twice again — 1s, 2s, 4s, 8s. Sprinkle in jitter (a random nudge) so a thousand workers that all failed at the same instant don't all retry at the same instant and stampede the recovering provider together.
Retries can't go on indefinitely. After a capped number of attempts you give up gracefully: the message moves to a dead-letter queue (DLQ) — a side queue for messages that exhausted their retries. Nothing is lost; a human or an automated job can inspect the DLQ, fix the cause (bad token, malformed payload), and replay. Underpinning all of it is a persisted notification log: a durable record of every notification and its status (queued → sent → delivered / failed). The log is what makes retries safe to resume after a crash and gives you the audit trail to answer "did user 42 ever get this?"
At-least-once delivery and duplicate events both lead to the same place: the same push arriving twice. Maybe a worker crashed after sending but before recording success, so the message reappears. Maybe the upstream service fired the "order shipped" event twice. Either way the user gets buzzed twice for one thing, and that erodes trust fast.
The fix is an idempotency key — also called a dedup key. Every notification carries a stable ID derived from the event, not from the attempt: something like order-shipped:order-998:user-42. Before sending, the worker checks a fast store (Redis with a TTL works well): have I already sent this key? If yes, drop it silently — the duplicate does no harm. If no, record the key and send. Now "deliver at least once" effectively becomes "deliver exactly once as observed by the user," which is what people actually care about.
The key has to be derived from the event, never freshly generated per attempt — a per-attempt key would be unique every time and dedup nothing. This is the same idempotency principle as Chapter 1's at-least-once queues: make processing the same message twice a no-op, and at-least-once stops being scary.
send(k) applied twice equals send(k) applied once. Idempotence is to side-effecting actions what a set is to a list: order and repetition stop mattering. That's exactly what lets you retry fearlessly.Two very different things can go wrong with volume, and you need a limiter for each.
The mechanism is the token bucket from Chapter 5: a bucket refills with tokens at a steady rate, each send spends one, and when the bucket is empty, sends are throttled until it refills. Its burst tolerance is perfect here — a saved-up bucket lets a legitimate flurry through, while a sustained flood gets smoothed down to the refill rate. You run one bucket per user (small, in Redis) and one per provider (shared across all your workers, so twenty workers can't collectively blow a quota one worker would have respected).
You don't hand-write every message. You write a template once — "{name}, your order {orderId} has shipped!" — and the system fills in the blanks. This parameter substitution keeps content consistent, lets non-engineers edit copy, and means a wording change is one edit instead of a code deploy.
Three layers ride on top of plain substitution:
"shipped" becomes "expédié" by looking up the same template key in the right locale.Keep templates versioned and out of code. When marketing wants to tweak the wording, that's a content change, not an engineering ticket.
render : (Template, Params, Locale, Channel) → String. Same inputs, same output — a pure function. That purity is what lets you preview a notification, snapshot-test the copy, and render a million of them in parallel.The fastest way to make people hate your product is to notify them about things they didn't ask for, on channels they didn't choose, at hours they're asleep. So every user gets a preference center: a grid of channel × category switches. They can take marketing email but not marketing push, keep security alerts on every channel, and turn off the "someone liked your post" buzz entirely. The notification server consults these settings before fanout — an opted-out channel is simply never enqueued.
Quiet hours are a special preference: hold non-urgent notifications during the user's night and release them in the morning (or drop them if they've gone stale). Urgent ones — a security alert, a fraud warning — bypass quiet hours by category.
And this isn't just courtesy; it's the law, and the laws differ by channel:
Practically, this means a one-click unsubscribe path on every channel, consent stored as data you can prove, and opt-out checks that are impossible to bypass — they live in the core send path, not bolted on at the edge.
A notification system fails quietly — nothing crashes, messages just stop arriving — so observability is the whole game. Watch these:
To scale, the notification servers are stateless, so you shard them horizontally behind a load balancer — partition by user so a user's preferences and rate-limit bucket stay coherent. Each channel scales independently by adding workers to its pool. And because every channel is a third party that will have outages, plan provider failover: a secondary SMS provider you can flip to when the primary degrades, health checks that detect the degradation, and the persisted log so in-flight notifications survive the switch.
Here's the whole system as a checklist — the order in which one event becomes many delivered notifications:
The heart of the system — who gets what, on which channel, rendered how — is pure decision-making. No network, no clock, no database: just a notification plus the user's preferences in, and a list of ready-to-send messages out. The actual provider calls (APNs, Twilio, SendGrid) are effects that happen after this function returns, at the edge. Keeping dispatch pure means you can unit-test every routing and opt-out rule without mocking a single network. Flip between the languages; the shape is identical.
dispatch never touches the wire — it returns a plan (a list of ChannelMsg), and a thin effectful layer carries that plan to the providers. Routing and filtering are pure; the I/O lives at the boundary. That's the same lesson as every other chapter: push effects to the edges and the core stays testable, replayable, and easy to reason about.