Retrieval-augmented generation
Giving the model knowledge it doesn't have — and why citing sources changes what 'trustworthy' means.
Pretraining knowledge is frozen and public. It knows nothing about your company's policies, this week's prices, or yesterday's news — and retraining per customer is impossible. Retrieval-augmented generation (RAG) fixes this by assembling the model's context at query time: retrieve relevant documents, paste them into the prompt, generate.
Start here
RAG = open-book exam. The model already knows how to read and reason; retrieval supplies the book. Same frozen weights, fresh knowledge.
The loop, made visible
Run real queries against a toy corpus — watch retrieval pick chunks, the top-k cut apply, and the answer cite its source. Then toggle rerank and watch the second-stage scorer rescue a relevant chunk the retriever missed:
RAG: retrieve relevant context, then generate from it
Toy corpus with keyword-overlap retrieval (real systems embed and search by vector similarity) — the loop shape is identical: query → retrieve top-k → stuff into prompt → generate with citations. Toggle rerank on: a second, query-aware scorer re-orders candidates — chunk 6 (about fees, no keyword overlap) jumps into top-k. Rerank scores here are fixed values standing in for a cross-encoder.
Four things the artifact shows that generalize to production:
- Retrieval is a ranking problem. Every chunk competes by relevance; the top-k boundary is a hard cut — chunk #2 at top-k 1 is invisible no matter how good it was. Real systems tune k like any precision/recall tradeoff.
- Retrieval quality caps answer quality. The generator can only reason over what retrieval found. Bad retrieval → confident nonsense or a missed answer; there is no prompt that conjures a missing document.
- Grounded failure is a feature. When nothing matches, the honest answer is "not in the documents." Products that force an answer instead convert absence into hallucination.
- Citations change the trust model. An answer with a source can be checked; an answer from memory is an appeal to authority.
How real retrieval works
The toy version matches keywords. Production retrieval embeds both the query and documents into vectors (often with a purpose-trained embedding model) and finds nearest neighbors by cosine similarity — an index (HNSW, IVF, etc.) makes millions of vectors searchable in milliseconds. Refinements layer on top:
- Chunking strategy — how documents are split (fixed windows, semantic sections) decides what a retrieved unit even is.
- Hybrid search — combine vector similarity with keyword/BM25; they fail differently, so the union is robust.
- Reranking — the retriever optimizes for speed (it scores every chunk in the index), so it's deliberately shallow. A reranker — typically a cross-encoder that reads the query and the candidate document together, rather than comparing two separately-computed vectors — re-scores just the top candidates, slow but far more accurate per decision. Standard pipeline: retrieve top ~50 fast, rerank to top ~5 accurately. Cross-encoders don't scale to a whole corpus, which is exactly why the two-stage split exists.
- Query rewriting — the LLM expands or decomposes the user's question before searching.
Note
RAG and fine-tuning answer different questions. RAG adds facts; fine-tuning adds behavior. For "make the model know our policy docs," retrieve. For "make the model write in our product's voice," tune. Confusing the two is a classic, expensive mistake.
This lesson has exercises attached — choosing the right top-k for a query, and spotting a prompt that conflates retrieved facts with model knowledge — once the exercises layer ships.
Next: hand the model tools instead of documents, and it stops answering and starts acting.