What Is Retrieval-Augmented Generation (RAG) and When Do You Need It?

What is retrieval-augmented generation, and do you actually need it? It’s the standard answer to “make the model answer questions about my documents”, and it’s reached for reflexively in cases where something simpler would work better.

The mechanism is easy to explain. Knowing when to skip it is the harder and more valuable part.

The problem it solves

A model knows what was in its training data. It doesn’t know your internal wiki, last week’s support tickets, or a contract you were handed this morning.

Ask about any of those and you get a confident answer assembled from general knowledge, which is to say a wrong one. The model has no way to signal that it’s guessing.

The original technique was set out by Patrick Lewis and colleagues in Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, which defined it as models that “combine pre-trained parametric and non-parametric memory for language generation”.

Parametric memory is what the model learned. Non-parametric memory is a searchable index you control. RAG is the plumbing between them.

The original implementation paired a sequence-to-sequence model with “a dense vector index of Wikipedia, accessed with a pre-trained neural retriever”, and set state of the art on three open-domain question answering tasks.

The reason it mattered was not the benchmark. It was that knowledge could be updated by changing the index rather than by retraining anything, which is the property everyone actually wanted.

How retrieval actually works

Ahead of time you split documents into chunks and convert each to a vector, a list of numbers positioning that text in a space where similar meanings sit close together.

At query time you embed the question the same way, find the nearest stored vectors, and paste the matching text into the prompt. The model answers from text it can see.

Retrieval matches on semantic similarity, not on correctness. A chunk discussing your topic in the wrong context scores just as well as the right one.

That single property explains most RAG failures. The system is confident because it retrieved something, and relevance is not the same as being right.

Chunking is where it quietly breaks

Chunking gets far less attention than the choice of vector database and causes far more problems.

Split too small and chunks lose the context that made them meaningful. A paragraph beginning “This approach fails when” is useless without knowing which approach. Split too large and the embedding averages across several topics, matching everything vaguely.

Practice	Why it helps
Split on headings, not character counts	Sections are natural boundaries; fixed splits cut sentences in half
Prepend document title and section	A fragment carries its own provenance
Overlap adjacent chunks slightly	Answers straddling a boundary are not lost
Add keyword search alongside vectors	Embeddings are poor at exact matches

Hybrid search deserves particular emphasis. Search for an error code or a surname and semantic similarity returns things about error codes rather than the specific one, so running keyword search in parallel is usually the single largest improvement available.

Position inside the prompt matters

Retrieving the right chunk isn’t sufficient, because where it lands in the prompt affects whether the model uses it.

Nelson Liu and colleagues at Stanford found in Lost in the Middle that “performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts”.

So retrieving twenty chunks and pasting them in order can bury the best one in the weakest position. Rank them, and put the strongest matches at the edges.

RAG versus long context versus fine-tuning

These solve different problems and get confused constantly, which leads teams to build retrieval systems they didn’t need.

Long context means putting the whole document in the prompt. If your corpus fits and you can afford the tokens, that’s simpler and more accurate, because there’s no retrieval step to get wrong. Our piece on what a large context window actually buys covers the limits.

The scale of what models absorb during training is part of why people reach for fine-tuning by mistake. The GPT-3 paper showed models handling new tasks from prompt examples alone “without any gradient updates or fine-tuning”, which suggests most adaptation problems are prompt or retrieval problems rather than training ones.

Fine-tuning teaches form, not facts. It’s a poor way to add knowledge, since facts learned that way can’t be updated without retraining and the model can’t cite where anything came from.

RAG earns its complexity when the corpus is far too large to fit, changes constantly, or when per-query cost matters at scale. At $6 to $30 per million output tokens, pasting a large document into every request adds up quickly.

A minimal setup that works

Start smaller than the tutorials suggest. Postgres with the pgvector extension handles millions of vectors comfortably, and you probably run Postgres already.

Chunk on headings with title and section prepended. Use any current embedding model, since differences between the good ones are small next to what chunking quality does. Retrieve with vectors and keywords together.

Then instruct the model to answer only from the retrieved chunks, cite which chunk each claim came from, and reply NOT FOUND if the answer isn’t there. That last instruction is not optional, and the reasoning behind it is covered in our prompting guide.

Measure the two failure modes separately

RAG systems fail in two distinct places, and teams routinely spend weeks fixing the wrong one.

Retrieval failure means the right chunk was never fetched. Test it directly: take thirty real questions, note which document holds each answer, and measure how often it appears in your top results. Below 80%, no amount of prompt work helps.

Generation failure means the chunk was fetched and the model still answered wrongly, usually by overriding the provided text with its own priors. That one is a prompting problem.

Reasoning models change this calculation slightly. DeepSeek’s R1 work reported models developing “self-reflection, verification, and dynamic strategy adaptation”, and a model that checks its own working is better at noticing when retrieved context does not actually answer the question.

When you don’t need it at all

If your knowledge base is a few dozen documents, put them in the prompt. If the answer lives in a database, write a query, because a model calling a SQL tool beats semantic search over exported text every time.

And if the question requires aggregation across many records, RAG is the wrong shape of tool entirely. Retrieving ten similar chunks cannot count anything.

Cost is worth checking before you build. Frontier training compute has grown 5x per year since 2020 per Epoch AI, and that pressure has pushed inference prices down steadily, which keeps shifting the break-even between retrieving and simply pasting.

The architecture underneath sets a hard limit on the alternative, though. Attention costs grow with sequence length, a property of the design described in Attention Is All You Need, so very long prompts stay expensive and slow no matter how cheap tokens get.

Large context windows changed this calculation without settling it, and our comparison of long context against retrieval covers where each still wins.

What would change this advice? Cheaper long context, mainly. If prompt caching and falling token prices make pasting a whole corpus routine, the case for retrieval narrows to genuinely large or fast-changing collections, which is a much smaller set of projects than currently build one.