When to fine-tune an AI model, and when not to bother

Should you fine-tune a model? Most teams that ask this should not, and the ones that should are usually solving a different problem than they think they are.

Fine-tuning teaches behaviour. It does not teach facts reliably, and confusing those two is where the money goes.

What fine tuning an AI model changes

You take a trained model and continue training it on your own examples, adjusting the weights so its outputs shift toward the pattern you showed it.

What shifts well is form: tone, structure, output format, domain vocabulary, and the shape of a good answer in your context.

Fine-tuning is how you teach a model to sound like your company. It is a poor way to teach it what your company knows.

Facts are the failure case. Knowledge encoded in weights can’t be updated, audited or cited, and when it’s wrong you have no way to see that it’s wrong until it appears in output.

The three options, side by side

	Prompting	Retrieval	Fine-tuning
Fixes	Instruction and format	Missing knowledge	Consistent behaviour
Time to try	Minutes	Days	Weeks
Update cost	Edit the text	Reindex a document	Retrain
Can cite sources	No	Yes	No
Per-request cost	Higher, long prompt	Higher, retrieved context	Lower, short prompt

The ordering is not arbitrary. Prompting is reversible in seconds and fine-tuning is not, so trying the cheap thing first costs you almost nothing when it works.

Prompting also does more than people expect. Google Research showed in Chain-of-Thought Prompting that eight worked examples reached state of the art on a maths benchmark, beating a fine-tuned model on the same task.

And for anything that changes, retrieval wins on structure alone. The original RAG paper framed the split as parametric versus non-parametric memory, and our guide to how retrieval works works through where each belongs.

Data quality beats data quantity, by a lot

The instinct is to gather as many examples as possible. The evidence points the other way.

Meta’s LIMA fine-tuned a 65B model on just 1,000 carefully curated prompts and responses, and reported that it produced responses “either equivalent or strictly preferred to GPT-4 in 43% of cases” in a human comparison.

A thousand good examples beat fifty thousand mediocre ones, because the model learns the average of what you show it. Inconsistent examples teach inconsistency, and one bad pattern repeated across a large set becomes a reliable behaviour.

So budget your effort accordingly. Curating examples is the work; running the training job is the easy part.

You almost certainly want LoRA

Full fine-tuning updates every weight, which needs serious hardware and produces a full-size copy of the model per variant.

Hu and colleagues at Microsoft introduced LoRA, which freezes the original weights and trains small rank-decomposition matrices instead. They reported it “reduces the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times” against full fine-tuning of GPT-3 175B.

Quantisation pushed that further. Dettmers and colleagues’ QLoRA reported finetuning a 65B model “on a single 48GB GPU while preserving full 16-bit finetuning task performance”, which put this within reach of a single machine.

The practical upshot is that adapters are small files. You can keep several, swap between them, and delete one that turns out badly, none of which is true of a full retrain.

Evaluate before you train

This is the step most teams skip and the one that determines whether the project succeeds.

Write your evaluation set first, before any training. Fifty to a hundred real inputs with the output you’d accept, held out entirely, and scored the same way every time.

Run the base model against it and record the number. Without that baseline you cannot tell whether fine-tuning helped, and eyeballing a handful of outputs reliably produces the conclusion you were hoping for.

Check general capability afterwards too. A model tuned hard on one task often gets worse at everything else, and that regression is invisible unless you test for it.

Smaller can beat bigger

The strongest argument for fine-tuning is economic rather than qualitative, and it’s underrated.

OpenAI’s InstructGPT work found outputs from a 1.3B tuned model “are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters”. A tuned small model that matches a large general one on your narrow task is dramatically cheaper to run.

That matters more as volume grows. With frontier output pricing spanning $6 to $30 per million tokens, a high-traffic feature can justify the tuning work on inference savings alone.

Shorter prompts compound the saving. A tuned model doesn’t need the long instruction block you were sending on every request, and that block was billed every time.

What it costs you later

A tuned model is a maintenance commitment. When the base model is upgraded you retrain, and your carefully curated dataset needs to still be valid at that point.

You also lose the ability to explain an output. Retrieval can show you the document it used; a tuned model can only produce the answer, which is a real problem in regulated work.

And training data becomes a governance surface. Whatever went into the examples is now inside the weights, which matters for both privacy and for the acquisition questions courts are currently asking about training corpora.

A reasonable sequence for fine tuning an AI model

Build the evaluation set. Try prompting properly, including examples in the prompt rather than instructions alone. Add retrieval if the gap is knowledge.

If a measurable gap remains after all that, and it’s about behaviour rather than facts, fine-tune with LoRA on a few hundred excellent examples and compare against your baseline.

There’s one common case that jumps the queue, and it’s worth naming. If you need structured output in a rigid format at high volume, tuning gets there faster than prompt engineering does, because format compliance is exactly the kind of behaviour weights learn well.

Even then, check whether your provider’s structured-output mode solves it first. Enforcing a schema during decoding is free and it cannot drift.

If you are still weighing this against the alternatives, our comparison of prompting, retrieval and fine-tuning reduces the decision to a single question about what the model is getting wrong.

Teams that follow that order rarely regret it. Teams that start at the end usually spend a month discovering their real problem was a badly written prompt.