Wiki
Core10 min read

Retrieval-augmented generation

Giving the model knowledge it doesn't have — and why citing sources changes what 'trustworthy' means.

Pretraining knowledge is frozen and public. It knows nothing about your company's policies, this week's prices, or yesterday's news — and retraining per customer is impossible. Retrieval-augmented generation (RAG) fixes this by assembling the model's context at query time: retrieve relevant documents, paste them into the prompt, generate.

Start here

RAG = open-book exam. The model already knows how to read and reason; retrieval supplies the book. Same frozen weights, fresh knowledge.

The loop, made visible

Run real queries against a toy corpus — watch retrieval pick chunks, the top-k cut apply, and the answer cite its source. Then toggle rerank and watch the second-stage scorer rescue a relevant chunk the retriever missed:

RAG: retrieve relevant context, then generate from it

#2
kw 2.00 · rr 0.95Tuition fees are ₹48,000 per year for grades 6–8, with a 10% sibling discount.
#1
kw 0.00 · rr 0.35Riverdale School was founded in 1998 by educator Mira Kapoor.
#3
kw 0.00 · rr 0.12The robotics club meets on Tuesdays after school in Lab 2.
#4
kw 0.00 · rr 0.18Bus routes cover 12 neighborhoods, with pickup at 7:20 AM.
#5
kw 0.00 · rr 0.08The annual sports day is held in December on the main field.
#6
kw 0.00 · rr 0.88Payments are handled at the accounts office; concessions apply yearly.
Answer: Tuition fees are ₹48,000 per year for grades 6–8 (10% sibling discount). [source: chunk 2]

Toy corpus with keyword-overlap retrieval (real systems embed and search by vector similarity) — the loop shape is identical: query → retrieve top-k → stuff into prompt → generate with citations. Toggle rerank on: a second, query-aware scorer re-orders candidates — chunk 6 (about fees, no keyword overlap) jumps into top-k. Rerank scores here are fixed values standing in for a cross-encoder.

Four things the artifact shows that generalize to production:

  1. Retrieval is a ranking problem. Every chunk competes by relevance; the top-k boundary is a hard cut — chunk #2 at top-k 1 is invisible no matter how good it was. Real systems tune k like any precision/recall tradeoff.
  2. Retrieval quality caps answer quality. The generator can only reason over what retrieval found. Bad retrieval → confident nonsense or a missed answer; there is no prompt that conjures a missing document.
  3. Grounded failure is a feature. When nothing matches, the honest answer is "not in the documents." Products that force an answer instead convert absence into hallucination.
  4. Citations change the trust model. An answer with a source can be checked; an answer from memory is an appeal to authority.

How real retrieval works

The toy version matches keywords. Production retrieval embeds both the query and documents into vectors (often with a purpose-trained embedding model) and finds nearest neighbors by cosine similarity — an index (HNSW, IVF, etc.) makes millions of vectors searchable in milliseconds. Refinements layer on top:

  • Chunking strategy — how documents are split (fixed windows, semantic sections) decides what a retrieved unit even is.
  • Hybrid search — combine vector similarity with keyword/BM25; they fail differently, so the union is robust.
  • Reranking — the retriever is deliberately shallow (often backed by an approximate-index that never even visits every chunk). A reranker — typically a cross-encoder that reads the query and the candidate document together, rather than comparing two separately-computed vectors — re-scores just the top candidates, slow but far more accurate per decision. Standard pipeline: retrieve top ~50 fast, rerank to top ~5 accurately. Cross-encoders don't scale to a whole corpus, which is exactly why the two-stage split exists.
  • Query rewriting — the LLM expands or decomposes the user's question before searching.

Note

RAG and fine-tuning answer different questions. RAG adds facts; fine-tuning adds behavior. For "make the model know our policy docs," retrieve. For "make the model write in our product's voice," tune. Confusing the two is a classic, expensive mistake.

This lesson has exercises attached — choosing the right top-k for a query, and spotting a prompt that conflates retrieved facts with model knowledge — once the exercises layer ships.

Check yourself

Eduspheria wiki · Large Language Models, Frontier

0 / 5 answered

  1. 1Why is a reranker used after retrieval?
    Multiple choice
  2. 2In the standard pipeline the lesson describes, the top 50 candidates are reranked down to how many?
    Numeric answer
  3. 3Retrieval quality caps answer quality: the generator can only reason over what retrieval found.
    True / false
  4. 4When nothing relevant is retrieved, what is the honest grounded answer the lesson recommends?
    Short answer
  5. 5Which problem does RAG solve versus fine-tuning?
    Multiple choice

Next: hand the model tools instead of documents, and it stops answering and starts acting.