Move money once,
prove it forever

Start with the happy path. A customer taps "Pay $20." That tap becomes a payment event — a small piece of data describing the intent ("charge $20 to this card, credit this merchant"). From there it travels a short relay:

• The payment service takes the event, validates it (is the amount sane? is this a known account?), and records that a payment has been requested — before anyone's card is touched.
• The payment executor is the part that actually does the money move. It talks to the outside world.
• The PSP — payment service provider, like Stripe or PayPal — is that outside world. It holds the card networks and bank rails. Your executor asks it to capture the funds; it answers succeeded, failed, or — the scary one — timed out, unknown.
• The ledger records the result as matching debit/credit lines (section 4). This is your source of truth.
• The wallet balance the user sees is just a view derived from the ledger — never edited directly.

Two directions matter. Pay-in is money coming into the system — a customer funding a purchase or topping up a wallet. Pay-out is money going out — paying a merchant, withdrawing to a bank. They share the same machinery but flip the accounts, and pay-outs are usually slower and more tightly controlled because money is leaving for good.

Networks lie. A client sends "charge $20," the charge actually succeeds, but the reply gets lost on the way back. The client sees a timeout, assumes failure, and — reasonably — retries. Without a defense, that second request charges the customer again. This is the single most common way to double-bill people, and it has a clean fix.

The client attaches an idempotency key to the request: a unique ID it generates once for this payment attempt and reuses on every retry (often a random UUID, e.g. pay_8f3c…). The rule on the server is simple: "if I've seen this key before, don't do the work again — just return the result I already produced."

Mechanically the server keeps an idempotency store — a table keyed by that ID. On each request it looks the key up:

• Key not found → this is the first time. Do the work, then save the result under that key before replying.
• Key found → this is a repeat. Skip all the work and return the stored result, byte-for-byte the same as the first reply.

So an idempotent operation is one you can safely repeat: doing it once or ten times leaves the world in the same state. The customer is charged exactly once no matter how many times the request arrives.

"Charge exactly once" sounds like a single guarantee, but you can't buy it directly — networks and crashes mean any message you send might be lost. What you can build cheaply is at-least-once: keep retrying until you get an acknowledgement, accepting that the work might therefore happen more than once. (Recall the queue from Chapter 1: at-least-once delivery means a worker may see the same message twice.)

The trick is the equation: at-least-once delivery + idempotent handling = effectively exactly-once. You retry freely so nothing is ever lost, and idempotency makes the duplicates harmless. The state of the world ends up as if the operation ran exactly one time.

This needs a guard at every boundary where a duplicate could sneak through:

• At the executor — before doing anything, check the idempotency key. Seen it? Return the stored result. This is dedup against retries inside your own system.
• At the PSP boundary — pass an idempotency key through to the provider too. Stripe and friends honor their own idempotency keys, so even if your executor retries the outbound call, the PSP captures the funds once.

Two layers of the same idea: dedup inside, dedup at the edge. Neither alone is enough — your executor might crash after calling the PSP but before recording it, so the PSP-side key is what saves you on the next attempt.

Here is the heart of the system. Money is never created or destroyed inside your platform — it only moves from one place to another. The accounting world encoded this 500 years ago as double-entry bookkeeping: every transaction is written as at least two lines that cancel out. One account is debited (money leaves it) and another is credited (the same money arrives), and the two sides are equal.

We model each line as an entry — an account and a signed amount (a delta). A transaction is a list of entries, and the iron rule is that the deltas sum to zero. A $20 pay-in might be:

• customer_card: −2000 (money leaves the card, in cents)
• merchant_wallet: +2000 (the same money lands in the wallet)

Sum: −2000 + 2000 = 0. Balanced. If a transaction's entries don't sum to zero, it's by definition wrong — money would have appeared or vanished — and you reject it before it's ever written.

Two more properties make the ledger trustworthy. It is append-only and immutable: you never edit or delete a posted entry; a mistake is fixed by appending a reversing transaction. And balances are derived, not stored as truth: an account's balance is just the sum of all its entries — a fold over the log. That means the books can always be re-derived and independently checked, which is exactly what auditors and regulators want.

Your ledger is internally perfect, and that is not enough. The PSP has its own records, the banks have theirs, and the only way to know your truth matches their truth is to compare them line by line. That comparison is reconciliation, and at scale it's a daily, automated job.

Each day the PSP hands you a settlement file — a list of the transactions it actually processed and the money it actually moved. Banks produce statements too. Reconciliation lines those up against your internal ledger and walks both sides:

• Matched — same transaction, same amount, on both sides. The overwhelming majority. Nothing to do.
• In your ledger, not in the PSP file — you think a payment succeeded that the PSP never settled. Maybe a timeout you optimistically recorded as success.
• In the PSP file, not in your ledger — the PSP moved money you didn't record (a missed callback, a dropped event).
• Both present, amounts differ — a fee, a partial capture, a currency rounding gap.

The job classifies each mismatch and routes it to a repair workflow: auto-fix the known-benign cases (post the missing fee entry), and queue the genuinely ambiguous ones for a human to investigate. Reconciliation is how you sleep at night — it's the safety net that catches the bugs every other layer missed.

Here's a trap that looks innocent. After recording a payment, the payment service needs to do two things: write the result to its database, and publish an event ("payment succeeded") so the wallet service, the notifications service, and analytics can react. Two writes, two different systems — the database and the message bus.

The problem: you can't make two separate systems commit atomically. If you write the DB and then crash before publishing, the payment is recorded but nobody downstream ever hears — the wallet balance never updates. Publish first and then crash before the DB commit, and you've announced a payment that didn't actually happen. This is the dual-write problem, and it's the same consistency tension from Chapter 9, now with money on the line.

The fixes all share one move — collapse two writes into one:

• Transactional outbox — in the same database transaction that records the payment, also insert the event into an outbox table. One commit, both or neither. A separate relay then reads the outbox and publishes to the bus, retrying until each event is sent. The atomic write is local; the publish is just at-least-once delivery on top.
• Event sourcing — make the event log be the database. State is the fold over events, so there's no second write to fall out of sync.
• Sagas — for a flow spanning several services, model it as a sequence of local transactions, each with a compensating action to undo it if a later step fails. No global lock; eventual consistency with explicit rollback.

A payment isn't an instant; it's a little journey with stages, and treating it as a state machine keeps it honest. A payment moves through explicit states — REQUESTED → PENDING → SUCCEEDED, or → FAILED — and only legal transitions are allowed. You can always answer "where is this payment right now?", and a stuck payment is visible rather than silently lost.

Because each step talks to slow, flaky external systems, the coordination is asynchronous and built to survive failure:

• Retries with backoff — a transient PSP error isn't fatal; retry, but wait longer each time (exponential backoff with jitter) so you don't hammer a struggling provider or stampede in lockstep.
• Dead-letter handling — a message that keeps failing after N attempts is moved to a dead-letter queue rather than retried forever. It's parked for inspection, and the rest of the pipeline keeps flowing.
• Eventual consistency downstream — the wallet balance, the email receipt, the analytics row catch up shortly after the payment is final. The ledger is the synchronous truth; the views converge a beat later.

The state machine plus idempotent steps means a crashed payment can simply be re-driven: read its current state, attempt the next legal transition, and any duplicate work is absorbed by the guards from sections 2–3.

The hardest failure in payments is not "it failed" — it's "I don't know." Your executor calls the PSP and the connection drops before the answer arrives. Did the money move or not? You must never guess. The discipline is: treat the outcome as unknown, record it as PENDING, and resolve it by querying the PSP for that idempotency key (or letting reconciliation catch it in the settlement file). Optimistically marking it succeeded is how you lose money; pessimistically marking it failed is how you charge twice on retry. Unknown stays unknown until the source of truth tells you.

Then there's the legal frame. Touching card data drags you into PCI-DSS, a strict standard for handling cardholder data. The winning strategy is scope minimization: handle as little card data as possible — ideally none. Let the PSP's hosted fields or tokenization capture the card so a raw card number never touches your servers; you store a token, not a PAN. Less data in scope means a smaller, cheaper, safer compliance surface.

Finally, everything leans on the audit trail: the append-only ledger plus immutable event log means every cent is traceable to who moved it, when, and why. That same immutability that made balances re-derivable is also what makes the system provable to an auditor — closing the loop on "move money once, prove it forever."

The whole chapter as a checklist — the rules that keep money safe:

1. 1Model the flow in layers — event → service → executor → PSP → ledger → wallet; pay-in and pay-out share machinery.
2. 2Idempotency keys — client mints one key per payment and replays it; the server returns the stored result on repeats.
3. 3Exactly-once = at-least-once + idempotent — dedup at the executor and pass a key through to the PSP.
4. 4Double-entry ledger — balanced debit/credit pairs, append-only, balances derived as a fold.
5. 5Reconcile daily — diff your ledger against PSP settlement files and bank statements; classify and repair mismatches.
6. 6Beat dual-writes — transactional outbox / event sourcing / sagas to keep state and events consistent.
7. 7Coordinate async — payment state machine, retries with backoff, dead-letter queues, eventual consistency downstream.
8. 8Reliability & compliance — unknown stays PENDING until queried/reconciled; minimize PCI scope; keep a full audit trail.

Two ideas from this chapter compress into a handful of lines. A transaction is a List[Entry] whose deltas must sum == 0 — that's the double-entry invariant. A balance is a fold over entries. And idempotency is a seen-set guard: apply no-ops if it has already processed that key. Flip the languages; the shape is identical.