🤖 System Design design chatgpt
🤖 Part V · AI-Era Systems · chapter 24 / 24

An LLM for millions,
token by token

A language model doesn't hand back a finished answer — it writes one word at a time, streaming each piece to you as it goes. This final chapter is about the machinery that makes that fast and affordable at scale: streaming, the KV cache, GPU batching and sharding, context windows, grounding, and cost.

Every chapter so far has been about serving data — rows, files, messages. This last one is about serving a computation: a giant neural network that, given some text, predicts what should come next. The product you know as ChatGPT is mostly the plumbing around that one prediction, repeated.

The surprising thing is how little of "serving an LLM" is new. The model is exotic, but the system around it is the same toolkit we've spent twenty-three chapters building: a request arrives, you stream a response, you cache what's expensive to recompute, you batch for throughput, you shard across machines, you queue when you're out of capacity, you rate-limit abuse. The model is just an unusually expensive, unusually slow backend — and everything you've learned about taming expensive, slow backends still applies.

We'll start from the single most important fact about an LLM — it produces its answer one piece at a time — and follow that fact through every layer of the system it forces into existence.

1What "serving an LLM" even means

Here's the whole loop in plain English. You send the model some text — your prompt. The model reads all of it and predicts one next word. Then it sticks that word onto the end of your text and reads the whole thing again to predict the next word. Then again, and again, one word at a time, until it decides it's done. Each of those pieces is a token — roughly a word or a chunk of one — and because each new token is generated from everything that came before it, this is called autoregressive generation ("auto" = self, "regressive" = feeding its own output back in).

That one fact reshapes everything. A normal web request computes an answer once and returns it. An LLM request runs the model hundreds of times for a single reply — once per token. So the thing you care about isn't "how long is the request," it's how long each token takes. Latency is measured per token, and a long answer is just many tokens in a row.

Two numbers define the experience. Time-to-first-token (TTFT) is how long you stare at a blank screen before anything appears — dominated by the model reading your whole prompt. Tokens per second is how fast the text then flows. Get TTFT low and tokens/sec high and the model feels alive; get either wrong and it feels broken, even if the final answer is identical.

λ
The FP framing: generation is an unfold. You start with a seed (the prompt) and a step function (the model: "given this context, what's the next token?"), and you repeatedly apply it, appending each result to the context. The whole answer is just that recurrence run to a stopping point — we'll write it as exactly that in the code section.

2Streaming tokens back to the client

If the model produces tokens one at a time, you shouldn't make the user wait for all of them. As soon as a token is ready, push it down to the browser and render it. That's why ChatGPT's answers type themselves out in front of you — you're watching the tokens arrive live, not a fake animation of a response that finished long ago.

The transport is a one-way stream from server to client. The simplest fit is Server-Sent Events (SSE): the client opens one long-lived HTTP connection and the server writes tokens into it as they're produced, one little message each. Where you need two-way traffic (interrupting mid-answer, voice), a WebSocket does the same job in both directions. Either way it's the callback / push idea from Chapter 10 — instead of the client polling "are you done yet?", the server calls back with each new piece the instant it exists.

Streaming barely changes the total time, but it transforms the perceived latency. A user who sees the first word in 300 ms feels they're being served, even if the full answer still takes ten seconds. So the metric everyone optimizes is time-to-first-token: get something — anything — on screen fast, then keep the stream flowing smoothly so it never visibly stalls.

Interactive · token streaming Watch the answer arrive one token at a time
prompt → "Explain a load balancer in one sentence."
30 tok/s
TTFT: — · 0 tok/s
time-to-first-token
tokens emitted
0
measured tok/s

3The KV cache — don't re-read the whole chat

Remember step one: to produce each new token, the model reads everything so far. Naively, that means generating the 500th token requires re-reading all 499 tokens before it — and the 501st re-reads all 500, and so on. The work per token would grow as the conversation grows, so a long answer would get slower and slower toward the end. That's unacceptable.

The fix is a cache. When the model reads a token, it computes two internal vectors for it — call them its key and its value (this is the machinery of "attention," where each new token looks back over the keys and values of all previous tokens). Those vectors only depend on the token and its position, so they never change once computed. So instead of recomputing them for the whole history every step, you compute them once per token and keep them around. That stash is the KV cache.

With the KV cache, each new token only has to do fresh work for itself and then glance at the cached keys/values of the past. The cost per token stays roughly flat instead of climbing with context length — turning a quadratic disaster into something linear and fast. The price is memory: the cache holds keys and values for every token in the context, so it grows with conversation length, and on a GPU that memory is precious and finite. Long contexts and many simultaneous users both eat KV-cache space, and running out of it is a real capacity limit.

Interactive · KV cache vs recompute Cost per token as the context grows
16 tok
cost/token: flat
work per new token
total to generate ctx
cache memory held
💡
The KV cache is the same instinct as Chapter 5's caching: "if you'll need this answer again, don't recompute it." The twist is that here the cached thing isn't a database row — it's the model's own partial work on the conversation so far.

4Batching for GPU throughput

A GPU is a machine built to do the same math on a lot of numbers at once. Hand it one user's token and most of its thousands of cores sit idle; hand it sixty-four users' tokens together and they all light up. So the single biggest lever for serving cost is batching: pack many users' requests into one pass through the model. Each pass costs about the same whether it carries one request or sixty-four, so throughput (tokens served per second across everyone) rockets up as the batch fills.

The naive version waits to collect a full batch, runs it, then collects the next — but LLM requests finish at wildly different times (one user wants three words, another wants three pages), so a fixed batch stalls waiting for the slowest member. The modern fix is continuous batching (also called in-flight batching): the moment any request in the batch finishes a token or completes entirely, a waiting request is slotted into the freed-up space. The batch is reshuffled every single step, so the GPU is never idling on stragglers.

But there's no free lunch — this is the throughput-versus-latency tradeoff in its purest form. A bigger batch means higher utilization and more total tokens per second, but each individual user's tokens come a little slower, because the GPU is splitting each pass across more people. Tiny batches give snappy per-user latency and waste the hardware; huge batches maximize tokens-per-dollar but make everyone wait their turn. Tuning that balance is the heart of LLM serving economics.

Interactive · batching & the throughput tradeoff Bigger batch → more total tokens/s, slower per user
8
GPU utilization
total throughput
per-request latency

5GPU inference & sharding the model

The biggest models don't fit on one GPU. A model's "weights" — the billions of numbers it learned during training — can total hundreds of gigabytes, far more than a single GPU's memory. When one box can't hold the thing, you do exactly what Chapter 8 taught for databases: you split it across machines. For models there are two ways to cut it.

  • Tensor parallelism — slice each layer's math across several GPUs, so they all work on the same token at the same time and combine their partial results. Great for speed, but the GPUs must talk constantly, so they need a very fast interconnect (and usually live in the same box).
  • Pipeline parallelism — put the first few layers on GPU A, the next few on GPU B, and so on, passing the token down the line like an assembly line. Less chatter between GPUs, but you have to keep the pipeline full or stages sit idle.

Real deployments combine both, plus the data parallelism of just running many copies. The hard operational truth underneath all of it: high-end GPUs are expensive and scarce. You can't just autoscale to infinity the way you do with cheap stateless web boxes — there may simply be no more GPUs to rent. So LLM serving leans hard on a queue (Chapter 7) in front of the GPU pool to absorb bursts, and on careful autoscaling that scales within whatever capacity you've actually reserved. When demand exceeds supply, requests wait — that's why you sometimes hit "at capacity" messages.

⚠️
The capacity wall: with ordinary web servers, scaling out is a billing decision. With GPUs it's a supply decision — the hardware is finite and contended. Queueing, admission control, and graceful "you're in line" behavior aren't optional niceties here; they're how you stay up when everyone shows up at once.

6Context windows & memory

The model can only look at so much text at once. That limit — the context window — is the maximum number of tokens (prompt + conversation history + the answer being generated) the model can hold in view at one time. It's finite for two reasons: the model was trained for a certain length, and the KV cache for a giant context would devour GPU memory. Run past it and the oldest tokens have to fall out of view.

So a long-running chat needs a memory strategy. The crude option is truncation: drop the oldest turns and keep the most recent ones that fit. Better is summarization: periodically compress old history into a short recap ("the user is debugging a Scala build; we tried X and Y") and carry that compact summary forward instead of the raw transcript. Either way the goal is to keep the most relevant tokens inside the window.

And what about facts the model never learned — your company's docs, today's news, a private wiki? You don't retrain the model for that; retraining (fine-tuning) is slow, expensive, and stale the moment the data changes. Instead you use RAG from Chapter 23: retrieve the relevant snippets from a search index at request time and paste them into the prompt, so the model answers from text it can see right now rather than from frozen training data. Memory becomes a retrieval problem, not a training problem.

λ
The framing: the context window is the model's only input — it has no hidden memory between requests, exactly like the stateless server of Chapter 1. Every "memory" feature (chat history, summaries, retrieved docs) is just something you chose to put into that input. The model is a pure function of its context.

7Grounding & guardrails

A language model predicts plausible text, and plausible isn't the same as true. When it confidently states something false, we call it a hallucination. The main defense is grounding: don't ask the model to recall facts from memory — give it the facts. RAG again does the heavy lifting; retrieve the source material, inject it, and instruct the model to answer only from it. Better still, have it produce citations pointing back at the retrieved snippets, so a wrong claim is visibly unsupported and a human can check the source.

Then there's safety. Both ends of the pipe need filters. On the input side you screen for prompts trying to extract harmful content or to "jailbreak" the model into ignoring its rules. On the output side you scan generated text before it reaches the user, blocking or rewriting responses that cross policy lines. These guardrails are often smaller, faster classifiers running alongside the big model — cheap to run, and they fail safe.

Finally, abuse is a systems problem, not just a model problem. Generation is expensive, so a flood of requests is both a cost attack and a denial-of-service. You lean on the same defenses as any API: rate limiting per user and per key (Chapter 5), quotas and tiers, anomaly detection for scripted abuse, and the ability to shed or slow traffic before the GPU pool melts. The model is special; protecting the endpoint in front of it is completely ordinary.

8Scale, cost & reliability

LLM serving has an unusual cost model: you pay roughly per token, and tokens are generated on hardware that's both expensive and in short supply. Almost every production trick is, at bottom, a way to serve a token for less money or to keep serving at all when capacity runs out.

  • Cache common prompts. Many requests share a long, identical prefix — a big system prompt, a fixed instruction block, the same retrieved document. Prefix caching keeps the KV cache for that shared prefix around so you compute it once and reuse it across users, skipping a huge chunk of work. For fully repeated requests, an ordinary response cache (Chapter 5) skips the model entirely.
  • Route by model size. Not every question needs the biggest, slowest model. Send easy or short requests to a small, cheap model and reserve the flagship for hard ones — model-tier routing that can cut cost dramatically with little quality loss.
  • Degrade gracefully under load. When the GPU pool is saturated, don't fall over — shorten max output lengths, drop to a smaller model, queue with an honest wait estimate, or shed the lowest-priority traffic. A slower or briefer answer beats an error.
  • Observe everything. Track TTFT, tokens/sec, queue depth, batch fill, KV-cache utilization, GPU memory, cost per request, and error/refusal rates. You can't tune the throughput-latency-cost triangle you can't see.

That's the whole system. Here's the arc of an LLM request, end to end — the order in which the pieces of this chapter fire for a single chat message:

  1. 1Accept & rate-limit — authenticate, check quota, push back abusive traffic before it costs a token.
  2. 2Build the context — assemble system prompt + trimmed/summarized history + retrieved (RAG) snippets, within the context window.
  3. 3Queue for a GPU — wait for capacity in the (possibly sharded) model pool; absorb bursts in the queue.
  4. 4Join a batch — slot into a continuous batch, reusing any cached prefix to skip recomputation.
  5. 5Generate autoregressively — one token per step, KV cache keeping each step cheap.
  6. 6Stream tokens out — push each token to the client over SSE/WebSocket; optimize time-to-first-token.
  7. 7Guard the output — run safety filters and attach citations before (or as) the text reaches the user.
  8. 8Meter & observe — record tokens, cost, and latency; feed it all back into routing and autoscaling.

9Generation as a fold over the context

Section 1 claimed generation is "fold the context to predict the next token, append, repeat." Here's that claim as code. The model is just the step function next : List[Tok] => Tok — give it the context so far, it gives back one token. Generation is the pure recurrence that keeps feeding the growing context back in. In Scala we write it as an explicit recursion that appends; in Haskell it's literally iterate / unfoldr — the unfold whose seed is the context and whose step is the model. Flip between them; the shape is identical.

λ
Generation is a pure recurrence with the model as the step function: fold the context → predict the next token → append → repeat. The KV cache is just the optimization that lets each step reuse the previous step's work instead of folding the whole context from scratch — an implementation detail under a beautifully simple shape.