RAG (retrieval-augmented generation)

RAG (retrieval-augmented generation) lets an AI assistant look things up before it answers. It first searches your own documents, such as product sheets or a help centre, and then writes the answer from the passages it found. A webshop chatbot can then give your actual return policy instead of a general guess. When a policy changes, you edit the document and the AI model itself stays as it is.

Where does the term come from?

The name comes from a 2020 paper by Lewis et al. at Facebook AI Research, UCL and NYU. They connected a language model to a searchable index of Wikipedia, and the combined model generated "more specific, diverse and factual language" than a comparable model without retrieval. The survey by Gao et al. (2023) uses the term for any large language model that draws on external sources while it answers.

How does it work?

Gao et al. describe the basic version in three steps:

1. Indexing: documents are split into chunks, an embedding model turns each chunk into an embedding (a list of numbers that captures its meaning), and these vectors are stored in a vector database.
2. Retrieval: the question is turned into a vector with the same model, and the database returns the chunks closest to it in meaning, a form of semantic search.
3. Generation: the question and the retrieved chunks go into the prompt together, and the model writes its answer from them.

Frameworks such as LlamaIndex and LangChain package these steps. What the survey calls "advanced RAG" adds work before and after retrieval: rewriting the query, combining keyword and semantic search, and reranking the chunks so the most relevant ones reach the model.

Why not train the knowledge into the model?

Fine-tuning changes the model itself through extra training. With RAG the knowledge stays in the documents. Lewis et al. showed the difference by giving their model an older Wikipedia index, from 2016 instead of 2018. Asked about 82 world leaders who had changed between those years, the model named the 2018 leader in 68% of cases with the 2018 index and in 4% with the 2016 index. Nothing was retrained.

For facts that change, such as prices or delivery terms, that is the practical gain: you edit a document instead of training a model. The costs move elsewhere. Changed documents have to be indexed again, and every question carries the retrieved passages as extra tokens.

Where does it go wrong?

RAG reduces hallucinations and does not end them. In a human evaluation by Lewis et al. on Jeopardy questions, raters found RAG more factual than the model without retrieval in 42.7% of 452 comparisons and less factual in 7.1%. Gao et al. list the weak spots that remain: retrieval can miss the passage that holds the answer or return irrelevant ones, and the model can still write things the retrieved text does not support. The answer is also only as good as the documents. An outdated PDF in the index produces an outdated answer.

Example

Suppose a webshop adds a chatbot to its help page. A customer asks: "Can I return a sale item?" The system searches the help centre, finds the returns article and the sale conditions, and passes both passages to the model with the question. The model answers from those passages and links to them. When the return period changes, the shop edits the help article, indexes it again, and the chatbot gives the new answer. If the search had returned only the general returns article, the model would have missed the exception for sale items.