AI scaling laws: has bigger stopped meaning better?

Have AI scaling laws hit a wall? The short answer is that the original laws haven’t broken. They’ve been corrected once, then partly sidestepped, and the thing everyone’s arguing about isn’t really the mathematics.

The curves that describe how models improve with more compute have held up across an enormous range. What changed is the recipe for spending a compute budget, and where the labs now spend it. Both shifts get reported as “scaling is dead”, and neither one is that.

AI scaling laws were narrower than people remember

In January 2020, Jared Kaplan and colleagues published Scaling Laws for Neural Language Models. They reported that “the loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude.”

The same team had already shown what scale alone could do. The GPT-3 paper reported a 175-billion-parameter model handling new tasks “without any gradient updates or fine-tuning”, which is what made the scaling question urgent in the first place.

Seven orders of magnitude is the part worth pausing on. That’s a relationship holding from tiny models to enormous ones without bending. Empirical results in machine learning rarely behave that well.

The paper also found that “other architectural details such as network width or depth have minimal effects within a wide range.” That finding shaped the following three years. If architecture barely mattered and scale did, the rational move was to scale.

But note what the paper actually concluded about how to spend a budget: “Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.”

Very large models, modest data. That recommendation turned out to be wrong.

The curves never broke. The recipe for spending a budget against them was corrected twice, and then deliberately ignored.

Finding	Year	What it changed
Kaplan scaling laws	2020	Big models, modest data
Chinchilla	2022	Double the model, double the tokens
Deployment economics	Since	Train small models far past compute-optimal
Inference-time reasoning	2025	Spend compute at request time instead

Chinchilla rewrote the recipe, and the correction was large

In March 2022, a DeepMind team led by Jordan Hoffmann published Training Compute-Optimal Large Language Models. Their opening line is blunt: “We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant.”

They didn’t argue this from theory. They trained “over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens” and read the answer off the results.

The conclusion: “for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled.”

Then they demonstrated it. Chinchilla has 70 billion parameters and was trained on four times the data of Gopher, a considerably larger model. It scored 67.5% on MMLU. An improvement of more than seven percentage points over Gopher.

Here’s what that means in practice. A smaller model, trained longer on more text, beat a bigger one trained the old way. The field had been buying parameters when it should have been buying tokens. That’s not a scaling law failing. It’s a scaling law being measured properly for the first time.

Then the labs deliberately ignored compute-optimal

Chinchilla answers one question: given a fixed budget to train a model, what’s the best split between size and data? That’s the wrong question if you’re going to serve the model to millions of people.

Training happens once. Inference happens forever. A smaller model trained past the compute-optimal point costs more up front and less on every request after that. For anyone running a product, the arithmetic favours overtraining.

Meta’s Llama 3 Herd of Models paper describes its flagship as “a dense Transformer with 405B parameters and a context window of up to 128K tokens”, released alongside much smaller models in the same family. Those smaller models are where the deployment economics live.

So the recipe changed twice. Kaplan said big models, modest data. Chinchilla said balance them. Deployment economics said train the small one for far longer than either suggests. None of those moves required the underlying curves to break.

The data ceiling has an actual date range attached

If tokens matter as much as parameters, the supply of tokens becomes a constraint. That’s the real limit in the room, and it’s been quantified.

Pablo Villalobos and colleagues asked the question directly in Will we run out of data?, revised in June 2024. They project that models “will be trained on datasets roughly equal in size to the available stock of public human text data between 2026 and 2032”. Earlier if models are overtrained, which as we’ve just seen is exactly what’s happening.

Read that carefully. It isn’t a claim that progress stops. It’s a claim that one specific input runs out on a schedule, and that the schedule is close enough to plan around.

Three responses are available: synthetic data, non-text data, or getting more out of the same tokens. All three are being tried. None has a settled answer yet.

Synthetic data is the most discussed and the least resolved. The worry is degradation across generations. A model trained on another model’s output inheriting and amplifying its errors. The counter-observation is that verified synthetic data, where outputs get checked before they’re used for training, behaves differently from unfiltered scraping of model output. Mathematics and code are the obvious cases, because correctness can be checked mechanically.

Non-text data is the less glamorous option and possibly the larger one. Video, audio and sensor data exist in quantities that dwarf public text. Whether they’re as informative per token for language ability is a genuinely open question.

The spending moved from training to thinking

The most substantive shift isn’t about pre-training at all. It’s about giving a model more compute at the moment you ask it a question.

DeepSeek’s DeepSeek-R1 paper, later published in Nature (volume 645, pages 633 to 638, 2025), reports reasoning ability emerging from reinforcement learning alone. The authors describe “pure reinforcement learning (RL), obviating the need for human-labeled reasoning trajectories”, with behaviours including “self-reflection, verification, and dynamic strategy adaptation” appearing without being demonstrated.

They report this beating “counterparts trained via conventional supervised learning on human demonstrations” on verifiable tasks, mathematics, coding competitions, STEM problems.

Why this matters for the scaling argument: it’s a second axis. You can spend more compute during training, or spend more during inference by letting the model work longer. When the first axis gets expensive, the second one absorbs investment. Progress continues, and the headline parameter counts stop being the story.

This has a direct cost consequence, and not a comfortable one. A reasoning model that thinks for thirty seconds costs meaningfully more per query than one that answers immediately. The savings from overtraining a small model can be given straight back at inference time.

Efficiency is quietly doing a lot of the work

One number gets left out of most versions of this argument, and it changes the shape of it.

Alongside the compute figures, Epoch AI reports “pre-training compute efficiency improving at roughly 3.0× per year”. Meaning the same performance can be reached with progressively less compute each year. Better architectures, better data curation, better training procedures all compound.

So the effective progress rate isn’t the 5× compute growth on its own. It’s compute growth multiplied by efficiency gains. A model trained today with the same hardware budget as one from a year ago starts from a meaningfully better position.

This cuts both ways in the debate. It means capability can keep improving even if compute spending flattens, which weakens the “we can’t afford the next 10×” argument. It also means some of what gets attributed to scale is really attributable to engineering, which weakens the claim that scale alone is the driver.

The wall argument deserves its best version

Everything above could be read as motivated defence of scaling. So here’s the strongest case on the other side, stated properly.

Loss is not capability. Scaling laws predict cross-entropy loss. How well a model predicts the next token. They don’t predict whether it can hold a coherent argument, avoid confident errors, or do a job end to end. A curve can keep descending smoothly while the things people actually want stop improving at the same rate.

Second, the growth rate is brutal. Epoch AI puts frontier training compute at “5× per year since 2020”, with the top-5 trend having “grown by a factor of ~10,000” over that period. Sustaining that requires roughly an order of magnitude more spending every 18 months or so. Efficiency gains of about 3.0× per year help, but they don’t close a 5× gap.

Third, the observed gap between recent frontier releases has felt smaller to many users than the gap between earlier generations. That’s a subjective read, and subjective reads are exactly how bubbles and busts both get talked into existence. It’s still what a lot of practitioners report.

Put together: the curves can hold while the economics stop justifying the next step, and the user-visible improvement per dollar can fall even as loss keeps dropping. You don’t need scaling laws to break for scaling to stop being the strategy.

What these numbers can’t tell you

Some honest limits on everything above.

Chinchilla’s 400-plus models topped out at 16 billion parameters. The conclusions are extrapolations to scales the experiment didn’t reach. That’s standard practice, and it’s still extrapolation.

The data projection depends on assumptions about what counts as usable text and how aggressively models are overtrained. Change either and the 2026 to 2032 range moves.

Most importantly, the frontier labs don’t publish their current numbers. Parameter counts, token counts and training compute for the newest closed models are not public. Everyone arguing about whether scaling has stalled, in both directions. Is reasoning from the last generation of published results and from what they can infer.

That’s a genuine epistemic problem, not a rhetorical dodge. The people with the data have commercial reasons to characterise it a particular way.

What would change the conclusion

A few things would settle this more cleanly than another round of argument.

A published training run showing loss departing from the predicted power law at frontier scale would be decisive evidence that something has genuinely broken. Nobody has shown that.

Sustained flattening on benchmarks that resist contamination. Held-out sets the labs can’t have trained on. Would suggest capability has decoupled from loss. Watch those rather than the headline scores.

If Epoch’s 5× annual compute growth breaks and stays broken for two years, that’s the economics answering the question regardless of what the curves do.

And if synthetic data is shown to lift performance past the human-text stock without degradation, the 2026 to 2032 ceiling stops mattering.

Until one of those lands, “has scaling hit a wall” is a question about spending decisions dressed up as a question about mathematics. The mathematics is doing fine. The bills are the interesting part.

If you want the mechanics underneath this, our explainer on context windows covers why longer inputs cost what they do, and the piece on retrieval-augmented generation covers the main alternative to putting everything in the model itself.