How RLHF works, and what it costs the model

How does a model that just predicts the next word end up being helpful? It doesn’t, on its own. Something has to be done to it afterwards, and that something is reinforcement learning from human feedback.

It’s the step that turned a text predictor into an assistant, and the evidence for how much it matters is unusually clear.

Pretraining alone gives you the wrong thing

A base model is trained to continue text. Ask it a question and a perfectly good continuation is another question, because that’s what often follows a question on the internet.

GPT-3 demonstrated that scale alone buys a lot. The Language Models are Few-Shot Learners paper showed 175 billion parameters handling new tasks from examples in the prompt, “without any gradient updates or fine-tuning”.

But capable is not the same as cooperative. The model would answer, ramble, refuse nothing, and had no notion that one response was more useful to a person than another.

Stage one: show it what good looks like

The first step is supervised fine-tuning. Humans write example answers to real prompts, and the model is trained on those pairs in the ordinary way.

This gets you a model that behaves roughly like an assistant. It’s also expensive and it caps out, because you can only write so many examples and they only cover so much ground.

The deeper problem is that writing a good answer is hard, while recognising one is easy. Stage two exploits exactly that asymmetry.

Stage two: train a model to predict human preference

Instead of asking people to write answers, you show them several model outputs for the same prompt and ask which they prefer. Ranking is quick, and annotators agree with each other far more often on ranking than on drafting.

Those comparisons train a separate reward model whose only job is to score a response the way a person would. It becomes a stand-in for human judgement that can be queried millions of times.

Nobody writes the rules for a good answer. The reward model infers them from thousands of small preferences, then applies them at a scale no human panel could.

Stage three: optimise against that score

Now the language model is tuned to produce responses the reward model rates highly, using a reinforcement learning algorithm. A penalty keeps it from drifting too far from the supervised model, because a policy left unconstrained finds ways to score well while producing nonsense.

That drift penalty is doing real work. Optimise hard enough against any learned score and the model starts exploiting the scorer rather than satisfying the preference behind it.

The result is startling on the numbers

OpenAI reported the outcome plainly in Training language models to follow instructions with human feedback: “outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters.”

Model	Parameters	Human preference
GPT-3	175B	Baseline
InstructGPT	1.3B	Preferred, at 100x fewer parameters

Sit with that ratio. A hundredfold reduction in size, and people still liked the smaller model’s answers more.

It reframes what scale buys you. Raw capability came from pretraining, but the usefulness people actually experience came from a comparatively cheap step at the end, which is why arguments about scaling often talk past what users notice.

Why this changed what labs spend money on

Pretraining budgets are enormous and getting worse. Epoch AI puts frontier training compute growth at 5x per year since 2020, with the top models having grown roughly ten-thousandfold over that period.

Against that, preference tuning is cheap. Tens of thousands of comparisons and a fine-tuning run cost a rounding error next to a frontier pretraining job, and the InstructGPT result says the payoff in perceived quality is large.

So the rational allocation shifted. DeepMind’s compute-optimal work told labs how to divide a training budget between parameters and tokens, and RLHF told them that a small slice spent after training moves the user-facing needle more than either.

That is the quiet reason every assistant now feels broadly similar. They converged on the same cheap final step.

The alignment tax is real

Making a model more agreeable costs something, and the field has a name for it.

Tuned models tend to hedge more, refuse more often than necessary, and converge on a recognisable house style. Some of that is deliberate safety work. Some is the reward model learning that cautious answers get rated well, because a hedge is rarely wrong.

You can also lose sharpness on tasks the preference data never covered. If annotators rarely ranked technical edge cases, nothing in the process protects performance there.

Whose preferences, exactly

This is the part that deserves more attention than it gets. A reward model encodes the judgements of the specific people who did the ranking, under the specific instructions they were given.

Those annotators are a small group, usually contracted, working to a rubric written by the lab. Their preferences become the model’s defaults on tone, formality, what counts as sensitive, and how much certainty to express.

None of that is sinister, and all of it is a choice rather than a discovery. When a model feels oddly corporate, this is usually where it came from.

There is a measurement problem hiding in here too. Preference is not accuracy, and a reward model trained on what people liked will reward confident, fluent, well-structured answers whether or not they are correct.

That’s a plausible mechanism for why models state wrong things so smoothly. Nothing in the preference signal distinguishes a fluent correct answer from a fluent incorrect one unless the annotator happened to know the difference.

Simpler alternatives are winning ground

The three-stage pipeline is fiddly. Training a reward model and then running reinforcement learning against it is unstable and expensive, so the field has been looking for shortcuts.

Direct preference methods skip the separate reward model, optimising the language model against the comparison data directly. Less machinery, similar results in many settings.

The more striking development removes human preference from parts of the process entirely. DeepSeek’s R1 paper reports reasoning ability emerging from reinforcement learning alone, “obviating the need for human-labeled reasoning trajectories”, with self-reflection and verification appearing unprompted.

That works because those tasks are verifiable. You can check a proof or run a test suite, so the reward can come from the world rather than from a person, which is a fundamentally sturdier signal.

Where RLHF does not reach

Verifiable rewards only exist for verifiable tasks. There is no test suite for “is this explanation clear” or “is this the right tone for a condolence note”, and for that enormous category human preference remains the only available signal.

Chain-of-thought prompting showed a related limit from the other direction. Google Research found that eight worked examples in a prompt could beat a fine-tuned model on maths, which means some of what looks like a training problem is really a prompting one, and our guide to prompting covers where that line sits.

The counter-case against RLHF

A serious objection runs like this. RLHF does not make a model more truthful or more capable. It makes the model better at producing output that a particular group of raters approved of, and those are different targets that happen to overlap.

On that reading, the InstructGPT result is less impressive than it looks. Of course a model tuned on human preference wins a human preference evaluation. The measure and the training objective are the same thing.

The response is that usefulness is genuinely what people want from an assistant, and there is no view from nowhere to optimise against instead. If the goal is a tool people can use, human judgement is the target, not a proxy for it.

Both positions are defensible. What settles it in practice is that base models are close to unusable for most people and tuned ones are not, whatever that says about the underlying capability.

What would change the picture

If verifiable-reward training keeps generalising beyond maths and code into open-ended work, the human feedback stage shrinks and the alignment tax with it. That is the direction of travel and it is not settled.

The counter-development would be evidence that models tuned without human preference are more capable and less usable, which would tell you the tax was buying something after all.

Worth remembering that almost none of this is disclosed for current commercial models. Annotator instructions, dataset size and reward model details are not published, so any claim about how a specific assistant was tuned is inference. The architecture is public. The behaviour training is not.