Search by meaning,
not keywords

A classic keyword index (the kind behind a WHERE body LIKE '%car%' query or a Lucene/Elasticsearch inverted index) is a lookup table from words to the documents that contain them. It's fast, exact, and has carried the web for decades. It also has a blind spot baked into its design: it only knows the words you literally typed.

• Synonyms slip through. "car" and "automobile", "doctor" and "physician", "laptop" and "notebook computer" — same idea, different letters, zero overlap.
• Phrasing matters too much. "how do I fix a flat tire" misses a document titled "repairing a punctured wheel" even though it's exactly the answer.
• Meaning is invisible. A keyword index can't tell that "the capital of France" and "Paris" are about the same thing, because it compares spelling, not sense.

People patch this with synonym lists, stemming, and fuzzy matching, but those are hand-built band-aids over a fundamental gap: the index doesn't understand. What we actually want is to rank results by how close in meaning they are to the query — and to get that, we need a way to measure the distance between meanings. That's exactly what the next section gives us.

An embedding is a list of numbers — a vector — that pins a piece of text to a point in space. A trained model reads "puppy" and emits something like [0.21, 0.88, -0.34, …] with hundreds or thousands of numbers. On its own that list is meaningless. What matters is the arrangement: the model is trained so that texts with similar meaning get vectors that point in nearly the same direction, and unrelated texts point all over the place.

So "dog" and "puppy" end up as nearby points; "dog" and "stock market" end up far apart. The same model can embed sentences, paragraphs, even images and audio — anything you can teach it to map into the shared space. Two images of beaches land near each other; a beach photo and a spreadsheet do not.

How do you measure "nearby"? Two common rulers:

• Cosine similarity — the angle between two vectors. Pointing the same way scores 1, perpendicular scores 0, opposite scores -1. It ignores length and cares only about direction, which is usually what "same meaning" means.
• Dot product — multiply matching coordinates and sum. If the vectors are normalized to length 1, the dot product is the cosine, which is why most vector databases normalize on the way in and just use dot products (they're fast).

The whole game from here on is geometric: store millions of these points, and answer a query by finding the points nearest to it.

Now the obvious plan: to answer a query, compute its similarity to every stored vector and keep the top few. This is exact (brute-force) nearest neighbor, and for a few thousand vectors it's perfect — fast, simple, exactly right.

It falls apart at scale. Imagine 100 million documents, each a 1,536-dimension vector. One query means 100 million dot products of 1,536 numbers each — over 150 billion multiply-adds per search. Do that for thousands of queries per second and you need a server farm just to answer one question. The cost grows linearly with both the number of vectors and their dimensionality, and high-dimensional space is genuinely hard: the usual tricks that speed up 2-D or 3-D nearest-neighbor (like k-d trees) stop helping once you have hundreds of dimensions. This is the famous curse of dimensionality.

So we make a deal. We give up the guarantee of finding the exact nearest neighbors and accept finding almost the nearest ones — say 98% of the right answers — in exchange for being 100× to 1000× faster. That trade is approximate nearest neighbor (ANN), and it's the foundation every vector database is built on.

An ANN index is a clever data structure that lets you skip almost all of the vectors and still land on the right neighbors. Two designs dominate.

HNSW — a navigable small-world graph

HNSW (Hierarchical Navigable Small World) connects each vector to a handful of its near neighbors, forming a graph you can walk. To search, you start at an entry point and greedily hop to whichever neighbor is closest to the query, again and again, until no neighbor is closer — like asking "who do you know that lives nearer to this address?" and following the chain. It's layered like a skip-list: sparse long-range links at the top get you into the right region in a few jumps, then dense links at the bottom refine to the exact neighborhood. A search touches only hundreds of nodes out of millions.

IVF — cluster, then search a few clusters

IVF (Inverted File index) first groups all vectors into clusters (say 4,096 of them) by running k-means, and remembers each cluster's center. To search, you find the few cluster centers nearest the query and scan only those clusters, ignoring the rest of the data entirely. A knob called nprobe sets how many clusters you check: more clusters means higher recall but more work.

Both expose the same recall-vs-latency knob. In HNSW it's efSearch (how wide a frontier you keep while hopping); in IVF it's nprobe. Turn it up for accuracy, down for speed. HNSW tends to give the best recall-per-millisecond but uses more memory; IVF is leaner and pairs beautifully with compression (product quantization) to fit billions of vectors in RAM.

You don't embed a whole 80-page PDF as one vector — a single point can't represent a document that covers ten topics, and you couldn't tell the model which paragraph the answer came from. Instead you chunk: split each document into bite-sized pieces (a few hundred words, often with a little overlap so a sentence cut in half still appears whole somewhere), and embed each chunk on its own.

The indexing pipeline that prepares your corpus looks like this:

1. 1Ingest & clean — pull text out of PDFs, HTML, transcripts; strip boilerplate.
2. 2Chunk — split into overlapping passages sized for the embedding model's context.
3. 3Embed — run each chunk through the model to get its vector (the slow, costly step).
4. 4Store — write vector + metadata + the original text into the vector DB; metadata holds the source URL, author, date, tags.
5. 5Index — build the ANN structure (HNSW graph or IVF clusters) over the vectors.

You store the original text alongside the vector because at query time you'll need to hand the actual words to the language model — the vector is only for finding the chunk, not for reading it. And documents change, so the pipeline must support updates: re-chunk and re-embed edited docs, delete vectors for removed ones. HNSW supports incremental inserts; IVF sometimes needs periodic rebuilds as clusters drift, which is a real operational cost at scale.

Pure vector search is great at meaning but oddly bad at exact tokens. Ask for the error code SQLSTATE 23505 or the product SKU A1-998X and the embedding shrugs — those strings have no "meaning" to cluster on, they just need to match. Keyword search nails them. The fix is to run both and combine.

Hybrid search runs the query through a keyword ranker (classically BM25, the workhorse scoring function behind Lucene/Elasticsearch) and the vector index, then merges the two result lists. A common merge is Reciprocal Rank Fusion: each document gets points based on its rank in each list (1st place is worth more than 10th), and you sum the points — no need to make the two very different score scales comparable.

Then comes re-ranking. The first stage casts a wide net cheaply — pull the top ~100 candidates from hybrid search. A second, slower, smarter model (a cross-encoder that reads the query and each candidate together, instead of comparing pre-computed vectors) re-scores just those 100 and keeps the top 5. You pay the expensive model only on a tiny shortlist, getting near-cross-encoder quality at near-vector-search speed.

A language model on its own answers from what it memorized during training. That memory is frozen at a cutoff date, knows nothing about your private documents, and — worst of all — when it doesn't know, it often makes something up that sounds right. That's hallucination.

Retrieval-Augmented Generation (RAG) fixes this by looking things up first and putting the facts in front of the model. The flow:

1. 1Embed the question — turn the user's query into a vector with the same model used for the documents.
2. 2Retrieve — ANN-search the vector DB for the top-k most similar chunks (often with hybrid + re-ranking from section 6).
3. 3Assemble the prompt — paste those chunks into the prompt as context: "Using only the passages below, answer the question. Passages: …"
4. 4Generate — the model writes an answer grounded in the supplied passages.
5. 5Cite — because you kept each chunk's source metadata, you can show "[1] docs/billing.md" links so the user can verify.

The model stops being a know-it-all and becomes a careful reader of the documents you handed it. Answers are grounded (less hallucination), current (just re-index when docs change — no retraining), and citable. RAG is how chatbots answer questions about your data.

A vector DB is still a database, so the scaling moves from the rest of this book come back — with a few vector-specific twists.

• Sharding the index. A billion vectors won't fit on one machine. Split them across shards, search every shard in parallel, and merge the top-k results (a scatter-gather). Each shard builds its own HNSW/IVF index over its slice.
• Metadata filtering. Real queries are "find similar chunks where tenant = 42 and date > 2025." Combining a filter with ANN is tricky — filter too late and you waste the search; filter too early and you wreck the graph's connectivity. Good vector DBs do filtered ANN that respects both.
• Freshness. New documents must become searchable quickly. Inserts into HNSW are cheap; large rebuilds for IVF are not, so systems often keep a small "hot" index for recent data and merge it into the big one periodically.
• Cost of embedding at scale. Embedding is the expensive step — re-embedding a billion chunks because you switched models is a budget event. Cache aggressively, batch requests, and pick your model deliberately (bigger embeddings cost more to compute, store, and search).
• Eval & recall monitoring. ANN recall silently degrades as data shifts. Keep a labeled set of query→correct-chunk pairs and continuously measure recall@k and answer quality, so you notice when retrieval rots before your users do.

Here's the whole chapter as one pipeline — text in, grounded answer out:

1. 1Chunk & embed the corpus into vectors + metadata.
2. 2Index with an ANN structure (HNSW or IVF), sharded across machines.
3. 3Embed the query and ANN-search for the top-k nearest chunks.
4. 4Hybrid + re-rank — fuse with BM25, re-score the shortlist.
5. 5Assemble the retrieved chunks into the LLM prompt as context.
6. 6Generate & cite — a grounded answer with sources.
7. 7Monitor recall, latency, and answer quality; re-index as data changes.

Strip away the indexes and the infrastructure and the core of every vector search is two tiny functions: measure how similar two vectors are, then sort the corpus by that similarity and keep the top k. An ANN index is "only" a way to do this without scoring every single vector — but the meaning of a search never changes. Flip between the languages; the shape is identical.