Long context vs RAG: which one you actually need

Million-token context windows were supposed to kill retrieval. Just paste everything in and let the model find what it needs. It hasn’t worked out that way, and the reasons are worth having straight.

Both approaches put your documents in front of a model. They differ in cost, in accuracy, and in what happens as your corpus grows, and the differences aren’t subtle.

The comparison

	Long context	Retrieval
Setup	None, paste and go	Chunking, embedding, an index
Cost per query	The whole document, every time	A few relevant chunks
Corpus limit	The window size, hard stop	Effectively none
Freshness	Whatever you pasted	Whatever is indexed
Accuracy on specifics	Degrades mid-document	Depends on retrieval quality
Can cite a source	Loosely	Precisely, by chunk

The positional problem is not fixed

The case against stuffing everything into context starts with a finding that keeps replicating.

Stanford’s Lost in the Middle found performance “significantly degrades when models must access relevant information in the middle of long contexts”, with accuracy highest at the beginning and end of the input.

A stated window is a capacity figure, not a usable-attention figure. The middle of a long input is measurably a worse place to be.

Windows have grown enormously since that paper and the shape of the problem has survived, which our explainer on what a million tokens changes covers. A fifty-page report pasted whole can be summarised with its middle twenty pages silently underweighted.

The economics are not close

Context is billed on every request, and long context is billed in full whether the model needed it or not.

Paste a 300,000-token corpus to answer one question and you’ve paid for 300,000 input tokens. Ask ten questions and you’ve paid ten times, and nothing about the corpus changed between them.

Retrieval pays a one-off indexing cost and then a few thousand tokens per query. At any real query volume the difference is not a percentage, it’s a multiple, and our piece on feature costs shows how that compounds in conversation loops.

The architecture is why. Attention cost grows with the square of sequence length, a property of the design set out in Attention Is All You Need, so doubling the context does considerably more than double the work.

Prompt caching narrows the gap for repeated identical prefixes, and it’s worth using. It doesn’t change the fundamental shape, because the corpus still has to fit and still has to be attended over.

Where long context genuinely wins

Retrieval fragments documents, and some questions are about the whole rather than the parts.

Chunk quality is what determines how badly that hurts, and good embeddings are half of it. Reimers and Gurevych’s Sentence-BERT made sentence-level similarity practical in the first place, and a mismatched embedding model will fragment meaning that a window would have preserved.

Compare these two contracts. Trace this character through the novel. Find the inconsistency between section two and section nine. Those need the full document present, because the answer lives in relationships between distant parts that no chunk contains.

Agent workloads are the awkward middle case. A long-running agent accumulates its own transcript, and research this year found that masking stale observations helps until it doesn’t, which is the same positional problem arriving from a different direction.

Long context also wins when the corpus is small and the questions are few. Setting up a retrieval pipeline to ask three questions of forty pages is engineering for its own sake; paste it and move on.

And it wins on setup cost every time, which matters for prototypes. The mistake is shipping the prototype’s architecture to production because it worked in the demo.

Where retrieval wins

Anything that grows, changes or needs auditing.

The original RAG paper framed the split as parametric versus non-parametric memory, and the non-parametric half has properties weights and windows both lack: you can update one document, delete one record, and show which chunk supported which claim.

Deletion matters legally now, which our piece on erasure obligations covers. An indexed record can be removed; a pasted corpus lives in logs and caches you’d have to chase.

Citation is the other production requirement long context handles badly. A model that read 400 pages can gesture at where an answer came from; a retrieval system can name the exact passage, and in regulated work that difference is the product.

They compose better than they compete

The strongest current systems use both: retrieval to select the right documents, and a large window to hold more of each selected document than older systems could.

Reranking is the piece that makes generous retrieval safe. Khattab and Zaharia’s ColBERT showed late-interaction scoring recovering much of the accuracy of full cross-attention cheaply, so you can fetch widely and then order precisely before anything reaches the window.

Bigger windows made retrieval better, not obsolete. You can now pass whole sections with their surrounding context instead of isolated paragraphs, which fixes the fragmentation problem that used to be retrieval’s main weakness.

Our guide to when you need RAG covers the pipeline side, and the short version of the combination is: retrieve generously, then let the window absorb the generosity.

Deciding for your case

Count three things: documents, queries per day, and how often the material changes. A static corpus under a few hundred pages with occasional questions: paste it. Anything that grows, changes or gets queried constantly: index it.

If you need citations or deletion, choose retrieval regardless of corpus size, because those are structural properties of the architecture rather than quality ones you can prompt your way toward.

And whichever you choose, test with questions whose answers you know sit in the middle of the material. That single check catches the failure mode that both approaches are quietest about.

Run the same test after any model upgrade, too. Positional behaviour is one of the properties that shifts silently between versions, and a pipeline tuned around one model’s attention profile can degrade without a single line of your code changing.