Prompting vs RAG vs fine-tuning: how to choose

Prompting, retrieval and fine-tuning get discussed as competing options. They fix different problems, and picking wrongly is the most expensive mistake available in this field.

The decision takes about five minutes once you know what to ask, and teams routinely spend a month instead.

The one question that decides it

Ask what the model is getting wrong, and be specific about it.

Doesn’t know something means retrieval. Doesn’t behave right means fine-tuning. Wasn’t told properly means prompting, and that’s most cases.

	Prompting	Retrieval	Fine-tuning
Fixes	Instructions and format	Missing knowledge	Consistent behaviour
Time to try	Minutes	Days	Weeks
Update cost	Edit the text	Reindex a document	Retrain
Can cite a source	No	Yes	No
Per-request cost	Higher, long prompt	Higher, retrieved context	Lower, short prompt
Reversible	Instantly	Yes	Not really

Prompting does more than people expect

The instinct to skip past prompting is understandable and usually wrong, because the ceiling is higher than it looks.

Google Research showed in Chain-of-Thought Prompting that eight worked examples reached state of the art on a maths benchmark, beating a fine-tuned model on the same task.

That’s a large effect from how you ask, achieved in an afternoon, and it’s fully reversible. Our guide to prompting techniques covers the ones that reliably help.

Structured output is the other underrated lever. Using a provider’s schema-enforcement mode fixes format problems that prompt wording never quite settles.

Retrieval is for anything that changes

If the gap is knowledge the model doesn’t have, retrieval is structurally the right answer rather than merely the cheaper one.

The original RAG paper framed it as combining “pre-trained parametric and non-parametric memory”, and the practical consequence is that facts stay in a place you can update, audit and delete.

That last property is now a compliance feature. Deletion from an index is a database operation; deletion from weights isn’t possible at all, which our piece on erasure and model weights covers.

Retrieval has its own failure mode. Similarity is not correctness, so a system can cite a real document that doesn’t support the claim, and that error is harder to catch than a fabrication.

More retrieved chunks won’t save you either. Stanford’s Lost in the Middle found accuracy degrades badly for information in the middle of a long input, so five good chunks beat twenty mediocre ones.

Fine-tuning is for form, not facts

Tuning shifts behaviour: tone, structure, output shape, domain vocabulary, the recognisable pattern of a good answer in your context.

What it does badly is knowledge. Facts encoded in weights can’t be updated, audited or cited, and when one is wrong you find out in production.

The economics can still justify it. OpenAI’s InstructGPT work found a 1.3B tuned model preferred over 175B GPT-3, and a small tuned model matching a large general one on your narrow task is dramatically cheaper to serve.

Data quality decides the outcome. Meta’s LIMA got strong results from 1,000 carefully curated examples, and a thousand good ones beat fifty thousand mediocre ones every time.

What tuning costs after you ship it

The training run is the cheap part now. LoRA made it accessible, with Hu and colleagues reporting it cut trainable parameters by 10,000 times against full fine-tuning of GPT-3.

The expensive part arrives later. A tuned model is a maintenance commitment: when the base model is upgraded you retrain, and your carefully curated dataset has to still be valid at that point.

That matters more now than it used to, because base models are replaced every few weeks. Our piece on the release cadence covers who absorbs that churn.

You also lose explainability. Retrieval can show you the document it used; a tuned model produces the answer and nothing else, which is a real problem in regulated work.

They combine, and usually should

Presenting these as alternatives is the framing error. Mature systems use all three, each for the thing it’s good at.

A support assistant might be tuned to answer in your house style, retrieve from your current documentation, and use a prompt that tells it to say it doesn’t know rather than guess.

Tuning also reduces prompt length, which reduces cost per request. That saving is real at volume and it stacks with retrieval rather than competing with it.

Order of operations matters when you combine them. Tune on examples that include retrieved context, or the tuned model will behave differently in production than it did in evaluation, because the shape of its input changed.

Four symptoms and their answers

Worked examples make the decision faster than any framework does.

The model invents a product feature you don’t have. That’s missing knowledge, so retrieval, and a prompt instructing it to say NOT FOUND when the answer isn’t in the retrieved material.

The answers are correct but too formal for your brand. That’s behaviour, so tuning, once you have a few hundred examples of the voice you want.

The JSON is malformed one time in twenty. That’s neither. Use structured output enforcement, which fixes it at the decoding layer rather than by asking nicely.

It’s right on simple questions and wrong on multi-step ones. That’s reasoning capability, which none of the three fixes. Change tier, or break the task into smaller checkable steps.

The sequence that works

Build an evaluation set first. Fifty to a hundred real inputs with the output you’d accept, held out, scored the same way every time.

Run the base model against it and write down the number. Without that baseline you can’t tell whether anything you do afterwards helped, and eyeballing outputs reliably produces the conclusion you were hoping for.

Then try prompting properly, add retrieval if the gap is knowledge, and fine-tune only if a measurable behaviour gap survives both. Our guide to when tuning is worth it covers that last step.

Teams that follow that order rarely regret it. Teams that start at the end usually spend a month discovering their real problem was a badly written prompt.