By Luis Arrais

Author

Follow Earth on Google

For researchers who need survey answers, buying them from AI is starting to look cheaper than asking people. The pitch is a digital twin: hand a language model hundreds of one person’s own answers, and it should reply the way that person would.

A team of 23 scientists compared the AI twins’ answers with those of the real people they were meant to copy. The twins often responded more like demographic groups than like the individuals themselves.

Based mostly at Columbia University, the team ran 19 new studies covering 164 separate questions. Some asked whether a person would share a misleading news article; others, whether they’d apply for a particular job. It’s a test of AI as a survey respondent, not of AI in general.

Olivier Toubia, a professor at Columbia Business School and one of the paper’s authors, said anyone building AI digital twins should stop grading them against nothing.

“Try to compare the answers of digital twins to those of their human counterparts. Decide on how you will measure accuracy before computing it,” Toubia told Earth.com.

Twins barely beat a blank prompt

Each twin was built by giving GPT-4.1 one person’s answers to more than 500 questions, drawn from a public dataset called Twin-2K-500. That’s about 128,000 characters of demographics, personality scores, cognitive tests and money decisions per person.

Then those 1,784 people took studies none of them had seen before, and their twins answered the identical questions. That’s the hard version of the test: a twin can’t recite an answer it was trained on.

On the study’s accuracy measure, where 1.0 means the twin picked exactly what its human picked, the fully trained twins scored 0.748. A persona given nothing but 14 demographic boxes scored 0.746.

All that personal history bought almost nothing. A prompt with no personal information landed at 0.734, and random answers landed at 0.629.

Ranking people went better. Correlation between twin and human answers averaged 0.197 across the 164 outcomes, against 0.145 for the demographics-only personas. Toubia’s team put that at roughly the strength of the link between height and intelligence.

Personal detail got flattened into an average

Two of the five distortions come out of that result. The first is that the twins didn’t vary enough. Their answers were more tightly bunched than human answers on 154 of the 164 outcomes.

Instead of copying each person’s distinctive answers, the twins tended to drift toward average responses.

The second is stereotyping, and the team measured it three ways at once. Full twins were closest to the demographics-only personas, farther from the bare prompt, and farthest of all from the actual humans.

The personal data moved each twin, but it moved toward a demographic average rather than toward a person. AI writing tools do something similar with characters, where prompts that don’t mention identity still produce familiar stereotypes.

Accuracy favored the comfortable

The third distortion is about who gets represented well. Twin accuracy wasn’t spread evenly across the panel, so the researchers looked for which personal characteristics predicted it.

Twins were more accurate for participants with more education and higher household income, and for people with moderate political views and moderate religious attendance.

A survey run on twins would hear the well-off clearly and everyone else less so.

Which distortion worries the authors?

The fourth distortion is ideological. Compared with their humans, twins were more likely to say people are fair and can be trusted. They were more accepting of hiring decisions made by algorithms and less bothered by online ad targeting.

They also underreported how much they used Netflix and TikTok. Asked to rate ideas for creativity, they marked down the ones they were told came from people.

Asked by Earth.com which of the five troubles him most, Toubia went straight to that one.

“If I had to pick one, it would probably be ideological biases. Because it has the potential to influence downstream decisions in subtle ways,” he said.

Subtle is the operative word. A twin warmer toward technology than its human is invisible in one answer, and AI systems can move opinions fast at scale.

Knowing too much to be human

The last distortion is hyper-rationality. On questions with objectively correct answers, the twins showed knowledge their humans didn’t have. On questions built to catch faulty reasoning, they reasoned better than the people they were copying, and they failed to reproduce two classic irrational shopping patterns.

Twins did not show a well-known default effect when the team tested it with an unpublished setup. They showed it strongly when the team used the famous published version.

The classic result was already sitting in the training corpus. The twin was reciting, not deciding, which is the weak spot behind AI systems that imitate moral reasoning without doing any, and behind models that stumble over human history.

Prediction has a ceiling

The main test was planned in advance, but the five distortions were patterns the researchers noticed afterward.

The twins also came from one U.S. panel and one family of models. The team varied the temperature setting and swapped in GPT-5, DeepSeek, Gemini and a fine-tuned GPT-4.1, and the best correlation any of it reached was 0.232.

Nothing here matched the bar the field actually cares about: how closely a person matches their own answers weeks later. Toubia thinks the training material matters more here than the model. Earth.com asked whether newer models are any less of a funhouse mirror.

“I’ve been doing research in which we train digital twins on past behavioral data, and results are more encouraging,” he said. “I’ve also been doing research where we train digital twins based on AI-moderated qualitative interviews, and the results are also more encouraging there.”

“Human behavior is just very hard to predict,” he said, and no volume of training data removes the “unpredictable variance” he expects to remain.

A 2024 study that built agents from two-hour life interviews reported friendlier numbers, and settling that will take a head-to-head test. For now, an AI digital twin is a well-informed guess about a category of person, not a copy of one. The same gap shows up wherever AI stands in for a person, including in therapy.

The full study was published in the journal Science Advances.