Saying "slow down"
at scale

Left unbounded, any open endpoint will eventually meet someone — a script, a scraper, a runaway retry loop, an actual attacker — who sends far more requests than you ever planned for. A rate limiter is the cap that keeps one caller from ruining the service for everyone else. It earns its place three ways:

• Abuse & denial-of-service protection. A login endpoint hit ten thousand times a second is either a credential-stuffing attack or a bug; either way you want to refuse it before it reaches your database. The limiter is your first, cheapest line of defense.
• Cost control. Every request you serve costs CPU, bandwidth, and — if it calls a paid third-party API or an LLM — real money. Capping requests caps the bill.
• Fairness across tenants. On a shared service, one greedy customer hammering the API can starve everyone else. Per-caller limits keep one tenant's spike from becoming everyone's outage. This is the "noisy neighbor" problem.

And it can live in several places, each a tradeoff between trust and reach:

• On the client — cheap and instant, but trivially bypassed (a hostile client just ignores it). Useful as a courtesy, never as a defense.
• At the API gateway / load balancer — the most common spot. A dedicated layer out front rejects excess traffic before it ever touches your application servers, so the rejection itself is cheap.
• In application middleware — inside your service, where you know exactly who the caller is and which expensive operation they're attempting. Closest to the business logic, but the request has already arrived.

Picture a bucket that slowly refills with tokens. Tokens drip in at a steady rate; each request that arrives must take one token to proceed. If a token is there, the request is allowed and the token is spent. If the bucket is empty, the request is rejected — come back later, once it has refilled. That's the whole idea, and it's called the token-bucket algorithm.

Two numbers define it. The refill rate R (tokens per second) is your long-run average allowance. The bucket size (capacity) is how many tokens can pile up while nobody's asking — and that's the clever part: it lets a caller who's been quiet save up a burst. If the bucket holds 10 and you haven't called in a while, ten requests can fire instantly; after that you're back to the steady drip of R per second.

That burst tolerance is exactly why it's the most widely used limiter (Stripe, AWS, and countless gateways run on it): it matches how real traffic behaves — bursty, then quiet — instead of forcing a rigid, evenly-spaced cadence. It's also tiny to store: just a token count and the timestamp of the last refill, per caller.

The token bucket's cousin turns the metaphor inside out. Instead of tokens dripping in, requests pour in and leak out through a small hole at a fixed rate. Incoming requests queue up; the bucket drains them in order at a constant pace; and if the queue is full, new requests overflow and are rejected. This is the leaking-bucket algorithm, and it's really just a fixed-size FIFO queue with a steady drain.

The defining difference is what comes out the other side. Where the token bucket lets a saved-up burst through all at once, the leaking bucket smooths any burst into a perfectly even stream — exactly R requests per second leave, no matter how spiky the arrivals were. That's its superpower and its cost in one:

• Use it when the thing downstream needs a steady, predictable feed — a payment processor that wants no spikes, a device that can only handle so many ops per second.
• Avoid it when bursts are legitimate and latency matters: a request that arrives during a lull still waits its turn in line, so a queued request can sit there adding delay even though the system is idle.

In short: token bucket permits burstiness; leaking bucket erases it. Same goal — bound the average rate — opposite personalities at the millisecond scale.

Here's the most obvious approach. Chop time into fixed slots — say one-minute windows — and keep a single counter per caller. Every request bumps the counter; when it reaches the limit, reject the rest; when the clock ticks over to the next minute, reset the counter to zero. One integer, one comparison. Beautifully cheap. This is the fixed-window counter.

It has one famous flaw, and it's worth seeing clearly. Because the window resets on a hard boundary, a caller can fire a full limit's worth of requests at the very end of one window and another full limit's worth at the very start of the next — and those two bursts sit only a moment apart. With a limit of 100 per minute, a caller can land 100 requests at 0:59 and another 100 at 1:00: 200 requests in about two seconds, twice the limit you thought you'd enforced. This is the boundary-burst problem.

For many uses that's tolerable — it's still bounded, just loosely. But if your limit exists to protect something fragile, "up to 2× in the worst case" may be exactly the case the attacker aims for. The next two algorithms exist to close that gap.

To kill the boundary-burst problem, stop using fixed slots and instead always look at the trailing window — the last 60 seconds counted from right now, not from a clock boundary. The way to do that precisely is to remember the timestamp of every request in a list, and on each new request, throw away the timestamps older than the window and count what's left. Under the limit? Accept and record this timestamp. Over? Reject. This is the sliding-window log.

It is exactly correct. There is no boundary to exploit because the window moves continuously with the clock, so a true rolling-60-second count never exceeds the limit. The catch is in the word "every": you store one timestamp per request, per caller. A caller doing 1,000 requests a minute costs you ~1,000 stored timestamps to police, and you pay that memory even for requests you ultimately reject. At scale, across millions of callers, that's a lot of bookkeeping for perfect accuracy.

What if you could get almost the accuracy of the log for almost the cost of the fixed window? You can, with a clever estimate. Keep just two counters — the count in the current fixed window and the count in the previous one — and blend them by how far you are into the current window. This is the sliding-window counter.

The formula is the whole trick. If you're 30% of the way into the current minute, you assume 70% of the previous window's requests still fall inside your rolling window, and add them to the current count:

estimate = current + previous × (1 − elapsed_fraction)

So with a limit of 100, a previous window of 80 and a current window of 30 while you're 30% in: 30 + 80 × 0.7 = 86 — under the limit, accepted. As the current window fills, the previous window's weight fades smoothly to zero, with no hard reset to exploit. It assumes the previous window's requests were spread evenly (they weren't, exactly), so it's an approximation — but it's a remarkably good one, off by a fraction of a percent in practice, while storing only two integers per caller. That blend of cheap and accurate is why it's the practical default for big public APIs.

Everything so far assumed one limiter holding the count. Real services run behind a load balancer with many identical servers, and the same caller's requests land on different ones. If each server keeps its own private counter, your "100 per minute" silently becomes "100 per minute per server" — twenty servers, and you've enforced 2,000. The fix is a shared store every server reads and writes — typically Redis, fast enough to consult on every request.

But sharing the count introduces a subtle, dangerous bug. Suppose the counter is at 99 and the limit is 100. Two servers, handling two requests at almost the same instant, both do the natural thing: read the count (both see 99), decide 99 < 100 so it's fine, increment, and accept. Now the counter says 101 and you've let through 101 requests against a limit of 100. This is the read-modify-write race, and it's the heart of distributed rate limiting.

The cure is to make the read-and-increment a single atomic operation that the store performs without interruption. Redis's INCR does exactly this: it increments and returns the new value in one indivisible step, so the two requests get back 100 and 101 respectively and the second one knows to reject. For anything more involved than a bare increment — say, the sliding-window-counter math — you bundle the whole read-compute-write into one atomic unit with a Lua script, which Redis runs to completion before touching anything else.

There's a final tradeoff. Consulting a shared store on every request adds latency and makes that store a dependency. Some high-throughput systems trade a little accuracy for speed: each server keeps a local counter and periodically syncs with the shared one. You can drift slightly over the limit between syncs, but you save a network round trip per request. Tighter sync, more accuracy and more cost; looser sync, the reverse — pick your point on that dial.

When the limiter says no, it should say it in a way the client can understand and act on. The standard is the HTTP status 429 Too Many Requests — not a generic error, but a specific "you're being throttled, this isn't a server bug." A good limiter pairs it with headers so a well-behaved client knows exactly when to try again:

• Retry-After — how many seconds to wait before retrying. A polite client honors this instead of hammering; an exponential backoff plus this hint is the gold standard.
• X-RateLimit-Limit — your total allowance in the window.
• X-RateLimit-Remaining — how many requests you have left right now.
• X-RateLimit-Reset — when the window resets (and your allowance refills).

These let clients self-regulate before they hit the wall, which is better for everyone — fewer wasted requests, fewer rejections to serve.

The last design decision is what happens when the limiter itself breaks — Redis is unreachable, the gateway can't reach its store. You must choose a default:

• Fail-open — if you can't check the limit, let the request through. Favors availability: a limiter outage doesn't take your service down, but a flood during that window goes unchecked.
• Fail-closed — if you can't check the limit, reject. Favors protection: nothing slips past, but a limiter outage now becomes a service outage for everyone.

There's no universal right answer — it depends on what the limit is protecting. For most public APIs, fail-open: a limiter is a guardrail, not a gate, and you'd rather degrade than go dark. For a limiter guarding something genuinely dangerous (a costly operation, a security-sensitive endpoint), fail-closed is the safer default. Decide deliberately; don't let your framework's default decide for you.

That's the whole picture. Here's the chapter as a checklist — the decisions you make when you design a limiter, in order:

1. 1Decide why & where — abuse, cost, or fairness; reject as early and cheaply as possible (gateway > middleware > client).
2. 2Pick an algorithm — token bucket for bursty traffic, leaking bucket to smooth, sliding-window counter as the accurate-and-cheap default.
3. 3Know fixed-window's flaw — the boundary burst can pass ~2× the limit; the sliding variants close it.
4. 4Mind the memory — the log is exact but stores a timestamp per request; the counter approximates it with two integers.
5. 5Share the count atomically — one Redis INCR or Lua script; never read-then-write across servers.
6. 6Speak 429 + Retry-After — and emit X-RateLimit-* so clients self-regulate.
7. 7Choose a fail mode — fail-open for availability, fail-closed for protection. Deliberately.

Section 2 claimed the bucket is "a tiny piece of state plus a pure update." Here's that claim in code. The entire algorithm is a value — a token count and a last-refill timestamp — and one function that takes the current time, refills by however much time has elapsed (clamped to capacity), then spends a token if there's one to spend. It returns the new bucket and a yes/no. No mutation, no globals; the clock is the only impurity, and it arrives as an argument. Flip between the languages — the shape is identical.