A friend offers to help you prepare for an interview. They ask the questions they think will come up, you answer, they push back, and by the end you are much better at answering those questions.
Then you sit the real interview and the first question is one neither of you imagined.
Nothing went wrong in the practice. The practice was useful. It was just made of what your friend could think of, and so were you by the end of it.
That is synthetic data. And it is becoming how most training data gets made.
Four waves
Where data comes from has changed three times already. It is worth laying them out, because each wave arrived when the one before it ran out of room, and each one moved the cost somewhere else.
Made. Data collected on purpose, for a question somebody had first. Surveys, experiments, instruments, the census. Small and expensive, and the purpose is recorded because it was designed in from the start.
Found. The scrape and the exhaust. Server logs, clicks, Common Crawl, ImageNet. Collected for one purpose and used for another, which is cheap, enormous, and the reason so much of it is quietly wrong for the job it ended up doing. That is a category is a decision at industrial scale.
Paid for. The annotation industry. Mechanical Turk, then labelling companies, then preference data for tuning. Human judgement bought by the hour.
Generated. Now.
The third wave is breaking on three fronts at once. Public human text is finite, and Epoch AI’s estimate of when the useful part runs out has already moved from the middle of this decade out towards 2028, which tells you what the estimate is worth and does not change the direction. The data that matters most is the data nobody can share, because it is medical, financial, or personal. And the regulatory cost of collecting and labelling keeps rising while the price of an annotator does not fall.
Something else happened alongside that. People can now produce things they never had the skill to produce. The same shift is reaching data itself, because a generator can be written by someone who understands the domain rather than someone who can implement a sampler. That changes who gets to build a dataset, and it is mostly good.
What actually changes
Here is the part I think is underrated.
When you collect data, reality does the guaranteeing. The rows are real whether or not you understand them, and your job is to clean them.
When you generate data, nothing guarantees anything. Your job is to be able to say why the thing you made behaves like the thing you could not get.
That is a measurement problem, and most of the field is still treating it as a generation problem. The hard part did not go away. It moved from cleaning to justification, and justification has no library.
Rare, or forbidden
Two industries got here before the language model people did, for opposite reasons.
Driving generated because the events that matter most are the ones you cannot collect. Waymo has driven somewhere around two hundred million autonomous miles on public roads and billions more in simulation, and its world model is used for things like a car crossing three lanes at once, a tornado, an elephant in the road. You cannot wait for those at any useful rate and you certainly cannot stage them.
Because they generated, they had to name the failure, and they did. The sim-to-real gap is a standing phrase in that field with people employed to measure it. Driving is honest about it because the gap kills people. Notice what the newer work actually does, too: the simulations are built out of real camera and lidar footage from the fleet, so the generator is anchored to measurement rather than to somebody’s idea of a road. That is the whole discipline in one design choice.
Banking generated for the opposite reason. Not because the records are rare, but because they are forbidden. Santander has open-sourced gen-fraud-graph so that graph-based fraud and money laundering models can be trained and benchmarked against realistic structures without touching real financial records. The mundane driver is worth saying out loud: a team can start building on day one instead of spending months in access review.
Their design rule is the interesting part. The generator never sees individual records, only aggregate statistics.
And they say the uncomfortable thing plainly, which is more than most: the more accurately a synthetic dataset reproduces the statistical structure of its source, the more useful it becomes, and the greater the risk that something about the source can be inferred from it. Fidelity and privacy pull against each other. That is a dial somebody has to set, not a problem anybody has solved.
Driving named its gap because the gap was fatal. Banking named its tradeoff because a regulator was going to ask. Both got a vocabulary by being accountable to something. The language model world is running the same play at larger scale without borrowing either.
Where it works, and why
The successes have something in common, and it is not model quality.
AlphaGeometry generated a hundred million synthetic theorems and proofs and used them to solve twenty five of thirty olympiad geometry problems, close to an average gold medallist, with no human demonstrations at all. That works because a geometry proof can be checked by a symbolic engine that does not care where the proof came from. Code is the same. It compiles or it does not. The tests pass or they do not.
This is the line I ended up drawing in human machine symbiosis, arriving from the other direction. What you can safely hand to a machine is not what it is good at, it is what you can check. Generated data obeys the same rule. Generate freely where a checker exists. Be careful where one does not.
The failure direction is documented too. Shumailov and colleagues showed in Nature that training recursively on generated data degrades a model, and the important detail is which part goes first. Not the average. The tails. The rare cases thin out and the middle of the distribution looks healthy the entire time, which is precisely why nobody notices until they need the rare case.
That is Braess’s paradox wearing new clothes. Adding a source can make the result worse, and more data is not monotonically better.
Designing the generator, not the rows
If the rows are downstream of the mechanism, the mechanism is the thing you are really building. Google’s Simula work is the most serious attempt I have seen to say so out loud. It treats dataset generation as a design problem rather than sample-level tweaking, and pulls apart what is usually one knob into three that can be set independently: coverage, complexity, and quality.
It also reasons from first principles rather than imitating a pile of seed data, which matters more than it sounds. A generator that imitates a sample can only ever reach as far as the sample did, so the gap it leaves is invisible by construction.
The finding I would keep from it is the negative one. There is no single best way to generate. Turning up complexity improved mathematics by around ten per cent and made legal reasoning worse. The settings do not transfer between domains, which means a generator is not a product you buy. It is something you design for the job and have to justify for the job.
The thing you cannot generate
In record linkage the most useful knowledge in the system is not the records. It is how often things occur.
That LONDON appears in an address corpus over a million times, and is therefore worth almost nothing as evidence. That a rare surname agreeing is worth a great deal, and a common one is worth very little. Those are not properties of the records. They are measurements of a population.
Generate the corpus and you generate the frequencies along with it. The rarity signal then becomes a statement about your generator rather than about the world, and a matcher trained on it will be confident in exactly the wrong places. It will have learned which names your sampler thinks are unusual.
You can synthesise the rows. You cannot synthesise how often the world does something. That has to be counted, and counting requires the thing you were trying to avoid needing.
Which is why Santander’s rule is the right shape, and worth reading as more than a privacy measure. Feed the generator aggregate statistics and never the records, and you have split the dataset honestly in two. The statistics were measured. The rows were made. A dataset stays trustworthy for exactly as long as everyone using it knows which half is which.
What to ask of a generated dataset
Not how much of it there is. Not how good it looks. Three questions:
- What mechanism produced it, and how would I know if it were wrong?
- What did the generator assume, and who chose that?
- What in here was measured, and what was invented?
The third one is the awkward one, because a generated row and a collected row look identical sitting in a table. Nothing in a CSV says which it was. A dataset that cannot tell you which half of itself came from the world is not a dataset with a quality problem. It is a dataset with a provenance problem, and those are the ones that stay wrong quietly.
Where I run out
- Can a generator ship its provenance with its output, so a row carries what made it, the way a page here carries a fingerprint? I do not know a format that does this and I suspect the reason is that nobody wants the answer recorded.
- If the tails go first, what is the cheapest tail test? Something you could run weekly, that fails loudly, and that does not require the real data you never had.
- If fidelity and privacy genuinely trade against each other, who sets that dial, and does the person in the source data ever find out where it was set?
- Driving named its gap because the gap was fatal. What has to go wrong, and how visibly, before a field names its own?
The honest position is that I find the direction persuasive and the enthusiasm premature. The interview practice was genuinely worth doing. It is just that afterwards you knew your friend’s imagination very well, and you still had not met the panel.
Reading
- Trinh, Wu, Le, He and Luong, Solving olympiad geometry without human demonstrations, Nature (2024). The clean case, and worth reading for how much of the work is the checker.
- Shumailov et al., AI models collapse when trained on recursively generated data, Nature (2024). The failure case, and the tails result specifically.
- Villalobos et al., Will we run out of data?, Epoch AI. Read the revisions as well as the estimate.
- The Waymo world model, for a generator that is deliberately anchored to measurement.
- Google Research, Designing synthetic datasets for the real world. Generation as design. Read it for the result that the settings do not transfer between domains.
- Santander, Realistic customer patterns without real customer data. Unusually straight about the fidelity and privacy tradeoff, from a party with every incentive not to be.
Related
- Human machine symbiosis, sixty years on, where the same verifiability line turns up in a different argument.
- A category is a decision, on data collected for one purpose and used for another.
- What an agent should refuse to do, on knowing when you are outside your own competence.
- Braess’s paradox.