JOURNAL / SEARCH & GENERATION

What is RAG?
Search first. Answer with evidence.

Retrieval-augmented generation connects a language model to a collection of sources. For each question, the system searches that collection and gives selected passages to the model as context for its answer.

Two archivists sort through folders and paper records in an archive room.RETRIEVAL / CONTEXT / ANSWER / SYMPLOKA
FINDING IS ONLY THE FIRST TEST

The answer starts with what the system brings into view.

A retriever narrows a collection. The answer still has to respect the sources it received.

01 / THE CORE IDEA

Give the model a route to material outside its weights.

A language model generates from patterns learned during training and from the context it receives now. Retrieval-augmented generation, or RAG, adds a search step: find passages related to the current question, place selected material in context, and ask the model to answer using it.

The foundational RAG paper describes the system as combining parametric memory—the model—with non-parametric memory—a searchable collection. In an application, that collection might contain product documentation, policies, manuals, research papers, or another set of sources the user is allowed to access.

RAG makes relevant evidence available to a model. It does not guarantee that the evidence is correct, that search found the best passage, or that the final answer follows it.

Updating a source collection can be faster than retraining a model, but new material must still be parsed, indexed, and made available to retrieval. The source can change while the answer pipeline still points to an older index.

02 / FOLLOW ONE QUESTION THROUGH

RAG is a chain of decisions around a search result.

  1. Choose and prepare sources. Decide which documents belong in the collection, extract their text, and keep identifiers, dates, versions, and access rules with them.
  2. Split material into retrievable passages. Boundaries should preserve enough context to make a passage useful without burying a short answer inside a huge document.
  3. Build an index. Search can use exact words and identifiers, dense vector representations, or a combination. A vector database is one implementation choice, not a requirement for every RAG system.
  4. Retrieve candidates for the question. Apply authorization rules, search the permitted collection, and optionally rerank candidates against the question.
  5. Assemble the context. Select a limited set of passages and provide source details that can be checked later.
  6. Generate and return an answer. The model interprets the supplied context. If the product shows citations, each one should point to the source passage that supports the nearby claim.

Each step can change the result. A relevant paragraph can be lost during extraction, split across a poor boundary, filtered out, ranked too low, or misunderstood after it reaches the model.

03 / SEARCH IS NOT ONE METHOD

Choose retrieval for the material and questions you have.

Keyword searchMatches words and terms. It can work well for names, codes, exact phrases, and technical identifiers.
Dense retrievalEmbeds questions and passages as vectors, then ranks candidates by a learned similarity. It can connect related wording, but similarity is not proof of relevance.
Hybrid searchCombines lexical and semantic results when both exact terms and paraphrases matter. The combined ranking needs to be measured on real questions.
RerankingRe-scores an initial shortlist against the question. It may improve ordering, but it adds latency and cost and cannot recover a useful passage that was never retrieved.

There is no universally best retriever. The Dense Passage Retrieval paper shows one successful learned approach on open-domain question-answering benchmarks; that result does not mean dense vectors beat keyword search for every corpus or product task.

For the representation behind vector search, read what AI embeddings represent—and what they leave out ↗.

04 / WHAT CAN GO WRONG

A retrieved passage can be relevant and still fail to support the answer.

Search misses evidenceThe answer is absent from the selected passages because of parsing, chunking, query wording, indexing, ranking, or filtering.
Sources disagree or expireA polished summary can hide a conflict, a superseded policy, or a fact that was never checked against the current source.
The model overreachesThe model may misread a passage, omit a qualification, combine unrelated snippets, or add a claim that the sources do not establish.
Retrieved text contains instructionsDocuments are data, not trusted policy. Treat instructions inside retrieved content as untrusted input, and keep authorization in application code.

A citation is a pointer, not a verdict. Check whether the cited passage actually supports the attached statement. For hostile or misleading retrieved content, see how prompt injection works ↗ and why permissions belong at the tool boundary ↗.

05 / WHEN RAG IS A FIT

Use it when external knowledge needs a clear update path.

RAG is worth evaluating when an answer depends on material that changes, belongs to a defined source collection, or needs to be inspected by a person. It is not a prerequisite for every chatbot. If the model already handles the task, the source collection is too weak, or the answer needs a structured database query, adding a retrieval pipeline may add complexity without improving the outcome.

Retrieval and fine-tuning change different parts of the system: retrieval supplies current context; fine-tuning adjusts model behavior. Compare them against the same task and evidence set in RAG or fine-tuning? ↗.

Before release, test whether the right passages are found, whether the answer is supported, whether the system abstains when the source collection is silent, and whether users can access only permitted documents. The detailed checklist is in how to evaluate a RAG system ↗.

PRIMARY RESEARCHLewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS (2020) ↗Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering, EMNLP (2020) ↗