Signum
Learn

RAG explained: how retrieval-augmented generation grounds AI answers

By Elliot Marsh, Editor · · 4 min read

Ask a general chatbot about your company's holiday policy and it will either refuse or make something up. It has never seen the document. Retrieval-augmented generation, usually shortened to RAG, is the standard fix: find the relevant passages first, then give them to the model and ask it to answer from them.

The short version

RAG has two steps:

  1. Retrieve. Search a collection of documents for the passages most relevant to the question.
  2. Generate. Put those passages into the model's prompt alongside the question, and ask the model to answer using them, ideally citing which passage said what.

The term comes from a 2020 paper by Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, which combined a neural retriever with a text generator. Today "RAG" is used loosely for any system that searches first and writes second.

Why not just train the model on the documents?

You can fine-tune a model on your data, but for most question-answering it is the wrong tool:

  • Freshness. Retraining every time a document changes is slow and expensive. With RAG you update the index.
  • Traceability. RAG can show the source passage for each claim. A fine-tuned model cannot point to where it learned something.
  • Access control. RAG can filter results by who is asking. Knowledge baked into model weights cannot be selectively hidden.
  • Hallucination. Models are better at reading facts from text in front of them than at recalling details from training.

Fine-tuning is better for teaching style, format or a specialised skill. Many systems use both.

How a typical RAG pipeline works

1. Preparing the documents (indexing)

  • Load documents: PDFs, web pages, tickets, wiki pages.
  • Chunk them into passages, often a few hundred words each, sometimes with overlap so sentences are not cut in half. Chunking by document structure (headings, sections) usually beats cutting at a fixed length.
  • Embed each chunk: an embedding model turns text into a list of numbers (a vector) that captures its meaning, so passages about similar things end up close together.
  • Store the vectors, the text and useful metadata (title, date, owner, permissions) in a vector database or a search engine that supports vectors.

2. Answering a question

  • Embed the question with the same embedding model.
  • Search for the chunks whose vectors are closest to the question's vector.
  • Optionally re-rank the top results with a more accurate but slower model that reads question and passage together.
  • Build the prompt: instructions, the retrieved passages (labelled so they can be cited) and the question.
  • Generate the answer, with citations.

Keyword search still matters

Vector search is good at meaning ("annual leave" matches "holiday entitlement"), but weak on exact strings such as product codes, error messages and names. Classic keyword ranking such as BM25 is strong exactly there. Many production systems run both and merge the results, which is called hybrid search.

Where RAG goes wrong

When a RAG system gives a bad answer, the cause is usually retrieval, not the model.

  • The right passage was never retrieved. Poor chunking, a weak embedding model or a question phrased differently from the documents.
  • Too much was retrieved. Stuffing in 30 passages adds noise. Research such as Lost in the Middle (Liu et al., 2023) found that models use information at the start and end of a long prompt better than information in the middle.
  • The documents disagree or are out of date. The model may pick the wrong version. Dates in metadata, and instructions to prefer the newest, help.
  • The question needs combining many sources. "How has our refund policy changed over five years?" needs several passages joined together, which simple top-k retrieval often misses.
  • The model ignores the context. It may answer from general knowledge instead of the passages. Clear instructions ("answer only from the sources; say if they do not contain the answer") reduce this.
  • Prompt injection. Retrieved text is untrusted. A document containing "ignore previous instructions" can influence the model. Keep instructions and retrieved content clearly separated, and never give a RAG system powers it does not need.

How to tell if your RAG works

Measure the two halves separately.

  • Retrieval quality: for a set of test questions with known answers, does the correct passage appear in the top results?
  • Answer quality: is the answer correct, is every claim supported by a cited passage, and does the system say "I don't know" when it should?

A small, honest test set of 50 to 100 real questions from users is worth more than any general benchmark.

Variations you will hear about

  • Agentic RAG: the model decides when and what to search, possibly several times, instead of one fixed search before answering.
  • GraphRAG: builds a graph of entities and relationships from the documents to answer questions that span many sources.
  • Long-context instead of RAG: with very large context windows, some teams paste whole documents in. That can work for small collections but costs more per question and does not scale to large libraries.

The bottom line

RAG is the simplest reliable way to make a language model answer from information it was not trained on. It is mostly a search problem: get the right passages in front of the model, tell it to stick to them and show its sources. When it fails, check the retrieval first.

Sources