A language model doesn't hand back a finished answer — it writes one word at a time, streaming each piece to you as it goes. This final chapter is about the machinery that makes that fast and affordable at scale: streaming, the KV cache, GPU batching and sharding, context windows, grounding, and cost.
Every chapter so far has been about serving data — rows, files, messages. This last one is about serving a computation: a giant neural network that, given some text, predicts what should come next. The product you know as ChatGPT is mostly the plumbing around that one prediction, repeated.
The surprising thing is how little of "serving an LLM" is new. The model is exotic, but the system around it is the same toolkit we've spent twenty-three chapters building: a request arrives, you stream a response, you cache what's expensive to recompute, you batch for throughput, you shard across machines, you queue when you're out of capacity, you rate-limit abuse. The model is just an unusually expensive, unusually slow backend — and everything you've learned about taming expensive, slow backends still applies.
We'll start from the single most important fact about an LLM — it produces its answer one piece at a time — and follow that fact through every layer of the system it forces into existence.
Here's the whole loop in plain English. You send the model some text — your prompt. The model reads all of it and predicts one next word. Then it sticks that word onto the end of your text and reads the whole thing again to predict the next word. Then again, and again, one word at a time, until it decides it's done. Each of those pieces is a token — roughly a word or a chunk of one — and because each new token is generated from everything that came before it, this is called autoregressive generation ("auto" = self, "regressive" = feeding its own output back in).
That one fact reshapes everything. A normal web request computes an answer once and returns it. An LLM request runs the model hundreds of times for a single reply — once per token. So the thing you care about isn't "how long is the request," it's how long each token takes. Latency is measured per token, and a long answer is just many tokens in a row.
Two numbers define the experience. Time-to-first-token (TTFT) is how long you stare at a blank screen before anything appears — dominated by the model reading your whole prompt. Tokens per second is how fast the text then flows. Get TTFT low and tokens/sec high and the model feels alive; get either wrong and it feels broken, even if the final answer is identical.
If the model produces tokens one at a time, you shouldn't make the user wait for all of them. As soon as a token is ready, push it down to the browser and render it. That's why ChatGPT's answers type themselves out in front of you — you're watching the tokens arrive live, not a fake animation of a response that finished long ago.
The transport is a one-way stream from server to client. The simplest fit is Server-Sent Events (SSE): the client opens one long-lived HTTP connection and the server writes tokens into it as they're produced, one little message each. Where you need two-way traffic (interrupting mid-answer, voice), a WebSocket does the same job in both directions. Either way it's the callback / push idea from Chapter 10 — instead of the client polling "are you done yet?", the server calls back with each new piece the instant it exists.
Streaming barely changes the total time, but it transforms the perceived latency. A user who sees the first word in 300 ms feels they're being served, even if the full answer still takes ten seconds. So the metric everyone optimizes is time-to-first-token: get something — anything — on screen fast, then keep the stream flowing smoothly so it never visibly stalls.
Remember step one: to produce each new token, the model reads everything so far. Naively, that means generating the 500th token requires re-reading all 499 tokens before it — and the 501st re-reads all 500, and so on. The work per token would grow as the conversation grows, so a long answer would get slower and slower toward the end. That's unacceptable.
The fix is a cache. When the model reads a token, it computes two internal vectors for it — call them its key and its value (this is the machinery of "attention," where each new token looks back over the keys and values of all previous tokens). Those vectors only depend on the token and its position, so they never change once computed. So instead of recomputing them for the whole history every step, you compute them once per token and keep them around. That stash is the KV cache.
With the KV cache, each new token only has to do fresh work for itself and then glance at the cached keys/values of the past. The cost per token stays roughly flat instead of climbing with context length — turning a quadratic disaster into something linear and fast. The price is memory: the cache holds keys and values for every token in the context, so it grows with conversation length, and on a GPU that memory is precious and finite. Long contexts and many simultaneous users both eat KV-cache space, and running out of it is a real capacity limit.
A GPU is a machine built to do the same math on a lot of numbers at once. Hand it one user's token and most of its thousands of cores sit idle; hand it sixty-four users' tokens together and they all light up. So the single biggest lever for serving cost is batching: pack many users' requests into one pass through the model. Each pass costs about the same whether it carries one request or sixty-four, so throughput (tokens served per second across everyone) rockets up as the batch fills.
The naive version waits to collect a full batch, runs it, then collects the next — but LLM requests finish at wildly different times (one user wants three words, another wants three pages), so a fixed batch stalls waiting for the slowest member. The modern fix is continuous batching (also called in-flight batching): the moment any request in the batch finishes a token or completes entirely, a waiting request is slotted into the freed-up space. The batch is reshuffled every single step, so the GPU is never idling on stragglers.
But there's no free lunch — this is the throughput-versus-latency tradeoff in its purest form. A bigger batch means higher utilization and more total tokens per second, but each individual user's tokens come a little slower, because the GPU is splitting each pass across more people. Tiny batches give snappy per-user latency and waste the hardware; huge batches maximize tokens-per-dollar but make everyone wait their turn. Tuning that balance is the heart of LLM serving economics.
The biggest models don't fit on one GPU. A model's "weights" — the billions of numbers it learned during training — can total hundreds of gigabytes, far more than a single GPU's memory. When one box can't hold the thing, you do exactly what Chapter 8 taught for databases: you split it across machines. For models there are two ways to cut it.
Real deployments combine both, plus the data parallelism of just running many copies. The hard operational truth underneath all of it: high-end GPUs are expensive and scarce. You can't just autoscale to infinity the way you do with cheap stateless web boxes — there may simply be no more GPUs to rent. So LLM serving leans hard on a queue (Chapter 7) in front of the GPU pool to absorb bursts, and on careful autoscaling that scales within whatever capacity you've actually reserved. When demand exceeds supply, requests wait — that's why you sometimes hit "at capacity" messages.
The model can only look at so much text at once. That limit — the context window — is the maximum number of tokens (prompt + conversation history + the answer being generated) the model can hold in view at one time. It's finite for two reasons: the model was trained for a certain length, and the KV cache for a giant context would devour GPU memory. Run past it and the oldest tokens have to fall out of view.
So a long-running chat needs a memory strategy. The crude option is truncation: drop the oldest turns and keep the most recent ones that fit. Better is summarization: periodically compress old history into a short recap ("the user is debugging a Scala build; we tried X and Y") and carry that compact summary forward instead of the raw transcript. Either way the goal is to keep the most relevant tokens inside the window.
And what about facts the model never learned — your company's docs, today's news, a private wiki? You don't retrain the model for that; retraining (fine-tuning) is slow, expensive, and stale the moment the data changes. Instead you use RAG from Chapter 23: retrieve the relevant snippets from a search index at request time and paste them into the prompt, so the model answers from text it can see right now rather than from frozen training data. Memory becomes a retrieval problem, not a training problem.
A language model predicts plausible text, and plausible isn't the same as true. When it confidently states something false, we call it a hallucination. The main defense is grounding: don't ask the model to recall facts from memory — give it the facts. RAG again does the heavy lifting; retrieve the source material, inject it, and instruct the model to answer only from it. Better still, have it produce citations pointing back at the retrieved snippets, so a wrong claim is visibly unsupported and a human can check the source.
Then there's safety. Both ends of the pipe need filters. On the input side you screen for prompts trying to extract harmful content or to "jailbreak" the model into ignoring its rules. On the output side you scan generated text before it reaches the user, blocking or rewriting responses that cross policy lines. These guardrails are often smaller, faster classifiers running alongside the big model — cheap to run, and they fail safe.
Finally, abuse is a systems problem, not just a model problem. Generation is expensive, so a flood of requests is both a cost attack and a denial-of-service. You lean on the same defenses as any API: rate limiting per user and per key (Chapter 5), quotas and tiers, anomaly detection for scripted abuse, and the ability to shed or slow traffic before the GPU pool melts. The model is special; protecting the endpoint in front of it is completely ordinary.
LLM serving has an unusual cost model: you pay roughly per token, and tokens are generated on hardware that's both expensive and in short supply. Almost every production trick is, at bottom, a way to serve a token for less money or to keep serving at all when capacity runs out.
That's the whole system. Here's the arc of an LLM request, end to end — the order in which the pieces of this chapter fire for a single chat message:
Section 1 claimed generation is "fold the context to predict the next token, append, repeat." Here's that claim as code. The model is just the step function next : List[Tok] => Tok — give it the context so far, it gives back one token. Generation is the pure recurrence that keeps feeding the growing context back in. In Scala we write it as an explicit recursion that appends; in Haskell it's literally iterate / unfoldr — the unfold whose seed is the context and whose step is the model. Flip between them; the shape is identical.