Test-time compute: paying for thinking instead of training

For a decade the way to get a better model was to train a bigger one. The last two years added a second lever, and it operates after training is finished.

Test-time compute means letting a model think for longer on a hard question. The core claim is that compute spent thinking can substitute for compute spent training, and it holds well enough to have reorganised the field.

Where the idea came from

The first evidence was almost accidental. Google Research showed in Chain-of-Thought Prompting that eight worked examples let a model reach state of the art on a maths benchmark, beating a fine-tuned model on the same task.

Nothing about the model changed. The prompt asked for intermediate steps, and the extra tokens did the work that additional parameters would otherwise have done.

Training compute is paid once and amortised across every request. Thinking compute is paid per request, forever. That is the whole trade.

Turning that from a prompting trick into a trained behaviour took reinforcement learning. A survey of reinforced reasoning with large language models traces how the field moved from asking politely to optimising for it directly.

Verifiable tasks are the boundary

This approach works spectacularly in some domains and barely at all in others, and the dividing line is unusually clean.

DeepSeek’s R1 work reported “self-reflection, verification, and dynamic strategy adaptation” emerging from reinforcement learning alone, on tasks where an answer can be checked mechanically.

Task	Gains from thinking longer	Why
Mathematics	Large	Answer is checkable, steps compound
Code that must run	Large	Tests provide ground truth
Logical puzzles	Large	Constraints can be verified
Summarising a document	Small	No verifiable target to search toward
Matters of taste	None	There is nothing to be right about

That table is a purchasing guide as much as a research summary. Paying reasoning prices on a summarisation workload buys latency and very little accuracy, which our comparison of when the extra cost pays works through with numbers.

Why preference training made this necessary

There’s a reason models needed to be taught to think rather than doing it naturally, and it goes back to how they were tuned.

OpenAI’s InstructGPT work found a 1.3B tuned model preferred over 175B GPT-3, because human raters reward answers that look confident and complete.

What raters don’t reward is visible uncertainty or a long working-out. So preference training pushed models toward answering immediately, which is exactly the behaviour that hurts on hard problems.

Reasoning training is partly an undo of that. It’s teaching a model that it’s allowed to take longer, which it had previously been trained out of.

What the reasoning trace is and is not

One common misreading is worth clearing up, because it changes how much you should trust what you see.

The visible chain of thought is generated text, not a log of the computation. It’s produced by the same process as the answer, and it can be wrong in the same ways.

So a plausible-looking trace leading to a wrong answer is entirely possible, and a correct answer can arrive after reasoning that doesn’t hold up when you read it.

What the trace genuinely gives you is a place to look when something goes wrong, which is more than a single opaque answer offers. Our explainer on what thinking out loud actually buys goes further into that distinction.

Efficiency became the research frontier

Once thinking became the product, thinking cheaply became the problem. Nobody wants to pay for a thousand reasoning tokens to answer a question that needed twenty.

Nvidia’s Llama-Nemotron work on efficient reasoning models is one answer, aimed at getting reasoning behaviour at a throughput that makes it deployable rather than demonstrable.

Distillation is the other lever. Train a smaller model on the traces a larger reasoning model produced, and much of that behaviour transfers at a fraction of the serving cost, which is how reasoning reached the mid-tier so quickly.

The other answer is routing. Send the easy 90% of requests to a fast model and reserve the reasoning tier for the cases that need it, which is a product decision rather than a research one.

The failure mode nobody warns you about

Longer thinking is not monotonically better, and that surprises people who assume more compute means more accuracy.

A model given room to reason on an easy question will sometimes talk itself out of a correct first answer. The extra steps introduce places to go wrong that a direct response never had.

Long reasoning traces also run into the same positional problem everything else does. Stanford’s Lost in the Middle found accuracy degrades for information sitting in the middle of a long context, and a model’s own reasoning becomes exactly that as it grows.

Which argues for matching the tier to the question rather than defaulting everything to the thinking model. That routing decision is worth more than most prompt tuning.

What it means for scaling

Test-time compute did not repeal the scaling laws. It gave the industry a second axis to spend on when the first one got expensive, which our explainer on the scaling debate covers.

It also changed who pays. A larger model shifts cost to the lab that trains it; a thinking model shifts cost to whoever runs the query, and that shows up in your bill rather than in a training budget you never see.

There’s a second-order effect worth noting. Because thinking tokens are billed like any other output, a reasoning model’s cost varies per question in a way a standard model’s doesn’t, so capacity planning gets harder.

Budget by worst case rather than average, or set an explicit reasoning cap. A single pathological question can otherwise consume what a hundred ordinary ones would.

Which is why the most useful question about any reasoning model is not how well it scores. It is how many tokens it burned getting there, and whether your task was one where those tokens could possibly have helped.