Synthetic training data: where it works and where it fails

What happens when AI labs run out of text to train on? That question stopped being hypothetical, because somebody put dates on it.

The proposed fix is synthetic data: models generating the training material for the next models. Whether that works is genuinely unsettled, and the arguments on both sides are stronger than the confident takes suggest.

The data wall has a date range

Pablo Villalobos and colleagues asked directly in Will we run out of data?, revised in June 2024. Their projection is that models “will be trained on datasets roughly equal in size to the available stock of public human text data between 2026 and 2032”.

Earlier if models are overtrained, which is exactly what labs now do for serving efficiency.

What makes this bite is the Chinchilla result. DeepMind found that for compute-optimal training, “for every doubling of model size the number of training tokens should also be doubled”, per Training Compute-Optimal Large Language Models.

Before Chinchilla, running out of text was a distant worry. After it, tokens became the thing you needed twice as much of every time you scaled.

And compute keeps climbing regardless. Epoch AI reports frontier training compute growing 5x per year since 2020, so the appetite for tokens grows with it.

What synthetic training data actually means

The term covers several quite different practices, and conflating them is why the debate goes in circles.

Type	How it is made	Risk
Verified generation	Model produces candidates, a checker keeps only correct ones	Low, correctness is established before training
Distillation	A strong model teaches a smaller one	Moderate, inherits the teacher’s errors and its ceiling
Augmentation	Existing text rewritten, translated or reformatted	Moderate, adds form without adding facts
Unfiltered scraping	Model output collected from the open web by accident	High, nobody chose it and nobody checked it

Only the last is obviously bad, and it is the one that gets used as an argument against the first.

Why the shortage arrived faster than expected

The original scaling work made data look like the easy input. Kaplan and colleagues found in Scaling Laws for Neural Language Models that “larger models are significantly more sample-efficient”, concluding that optimal training meant very large models on relatively modest data.

Follow that advice and text supply is not your constraint. Chinchilla overturned it two years later, and the token requirement jumped accordingly.

Then deployment economics pushed further still. Because inference cost dominates over a model’s life, labs train smaller models far past the compute-optimal point, which multiplies token demand again. Meta’s Llama 3 family is built around exactly that logic.

So the wall got closer through a sequence of individually sensible decisions, none of which was about data supply.

Where synthetic training data demonstrably works

Verified generation has the strongest evidence, and the reason is simple. In domains where correctness can be checked mechanically, you can throw away everything wrong before it ever reaches training.

Mathematics and code are the obvious cases. A proof either checks or it does not, a test suite either passes or it does not, and the filter costs almost nothing to run.

DeepSeek’s R1 paper is the clearest demonstration of the principle, reporting reasoning ability emerging from reinforcement learning alone, “obviating the need for human-labeled reasoning trajectories”. The model generated its own attempts and the verifiable reward sorted them.

That is synthetic data in everything but name, and it worked because the checking was cheap and reliable.

The same pattern shows up in how few examples a model needs when the format is right. Google Research found that eight worked examples could lift a model to state of the art on a maths benchmark, which suggests the binding constraint is often the quality and structure of data rather than the raw quantity.

That matters here because it means a smaller volume of verified synthetic examples can outperform a much larger pile of scraped text on the same task.

The model collapse objection

The counter-argument has a memorable name and a real mechanism behind it.

Train a model on another model’s output, then train a third on the second’s, and errors compound while variety shrinks. Rare constructions get sampled less often each generation, so the distribution narrows toward whatever the previous model found most probable.

Repeat that enough times and you get a model fluent in a smaller and smaller slice of language, confidently wrong in exactly the places its ancestors were.

There is a second-order worry underneath that one. As more of the open web becomes model-generated, the ordinary crawl starts containing synthetic text whether a lab wants it or not, so the unfiltered case arrives by default rather than by choice.

Which is part of why provenance marking has become a regulatory priority. The EU rule that pushed Anthropic into watermarking Claude output was written for transparency, and it happens to make future training corpora easier to clean.

The mechanism is not disputed. What is disputed is whether it describes what labs actually do, because the failure case assumes nobody filters anything.

Filtering is the whole argument

If you generate a hundred candidate solutions and keep the three that verify, you have not degraded anything. You have used the model as a search process and used a checker as the judge.

Collapse is a property of unfiltered recursive training, not of synthetic data as such. That distinction does a lot of work and it is often lost.

Which sets up the real limitation, and it is a hard one. Filtering requires a checker, and most of what people want models to do has no checker.

Where there is no checker

There is no test suite for whether an explanation is clear, whether a summary kept the right emphasis, or whether an argument is fair to the side it disagrees with.

For that enormous category, synthetic data offers no verification, so you are back to trusting the generating model. And the generating model’s judgement is the thing you were trying to improve.

This is the same wall that preference tuning runs into, approached from a different direction. Verifiable rewards work beautifully where verification exists and not at all where it does not.

The options that are not synthetic

Worth remembering that generating more text is not the only response to a shortage of it.

Video, audio and sensor data exist in quantities that dwarf public text, and multimodal training makes them usable. Whether an hour of video teaches a model as much about language as an hour of reading is an open question, and probably the wrong question.

Licensed private corpora are the other route. Archives, publishers and enterprises hold enormous stores of high-quality text that never entered the public crawl, and much of the recent deal-making is labs buying access to exactly that.

Then there is getting more from the same tokens. Epoch AI reports pre-training compute efficiency improving roughly 3x per year, and efficiency gains reduce how much data a given capability level requires in the first place.

What the honest position looks like

Synthetic data is not a general solution to the data wall and it is not a dead end. It is a technique that works in proportion to how well you can check the output.

So expect continued rapid progress on maths, code and anything else with a mechanical grader, and slower progress on judgement, tone and open-ended reasoning where nobody can filter reliably.

That uneven pattern is already visible in how models improve, and it is a better predictor of the next two years than any argument about whether synthetic data works in general.

What this means if you are building

The lesson transfers down to ordinary fine-tuning work, where the same logic applies at a much smaller scale.

If you can write a checker for your task, generating training examples with a strong model and filtering them is a genuinely good strategy. You are not hoping the model is right, you are keeping only the cases where you confirmed it.

If you cannot write a checker, be much more cautious. Generated examples for subjective tasks will encode the generating model’s habits, and you will have taught your smaller model to imitate a larger model’s style rather than to do the job well.

A middle path works surprisingly often: generate candidates, have a person review a sample rather than all of it, and measure whether the reviewed subset behaves differently from the unreviewed one. If it does, your filter is doing the work and you need more of it.

What would change the conclusion

A published result showing sustained capability gains on unverifiable tasks from synthetic data alone would break the framing above, and would be the most important paper of the year.

Evidence of measurable degradation in a deployed frontier model traced to synthetic training would break it the other way.

Neither exists publicly, which is worth stating plainly. Labs do not disclose their data mixes, so almost every confident claim about how much synthetic data is in current models is guesswork dressed as analysis, a problem that also afflicts benchmark comparisons.