Retrieval-Augmented Generation (RAG)

Squares are facts in a small knowledge base; circles are questions. Click a question — it retrieves its nearest fact by embedding distance, exactly Chapter 1's cross-modal search, just within a single modality this time. The retrieved fact is what actually grounds the final answer.

Retrieval is nothing more than the nearest-neighbor search from Chapter 1, applied to a document collection instead of images:

• $q$ — the query: the question being asked, embedded into the same space as the documents.
• $d$ — one candidate document in the knowledge base.
• $\text{docs}$ — the full collection of documents being searched.
• $\text{retrieve}(q)$ — the document nearest to the query, found by nearest-neighbor search.
• $\text{answer}$ — the final answer: grounded in the retrieved document when retrieval is used, or a fixed generic prior when it isn't.

1. Generation is simplified to a lookup, on purpose

   Retrieval doesn't change how a model generates — it changes what it has in front of it when it does, and that's the point this simplification is meant to make visible.

2. A real system still generates fluent text

   It just conditions that generation on retrieved passages instead of — or alongside — its frozen training-time knowledge.

Toggle retrieval off for the same question. Without it, the model falls back on one fixed, generic answer — the single most common one it might have learned — regardless of what's actually being asked. With retrieval, the answer comes from whichever fact is actually nearest to this question.

Three questions, one ungrounded fallback answer ("Paris" — the most famous capital, and this toy model's fixed prior):

1. Capital of Japan — no retrieval

   The ungrounded answer is "Paris" — wrong. Japan's own fact sits at distance $0.36$, but without retrieval the model never looks.

2. Capital of Japan — with retrieval

   The nearest document is "capital of Japan," at distance $0.36$ — more than seven times closer than the next-nearest fact. Retrieval corrects the wrong prior to "Tokyo."

3. Capital of France — a coincidence

   Here the ungrounded prior ("Paris") happens to already be correct. Retrieval still finds the same right answer — but the fact that grounding and non-grounding agree here is luck, not evidence that skipping retrieval is safe.