A synthetic user will never cancel your interview, never ask for a gift card, and never tell you your concept is pointless. That last one is the whole issue. Teams are now running discovery against AI-generated personas because they are fast and free, and the research on what those personas actually return is consistent enough that a product manager should know it before a single synthetic transcript lands in a roadmap review.
This post works through the questions I hear most often when a team proposes swapping some of its customer conversations for a model.
What does a synthetic user actually give you?
A synthetic user is a large language model prompted to answer as a particular kind of person: “a 34-year-old operations manager at a mid-sized logistics firm who is frustrated with spreadsheet reporting.” You interview it, survey it, or show it a concept. It answers fluently, in character, in seconds.
What it gives you is the statistically likely answer for someone described that way, based on text the model was trained on. That is a useful thing to have. It is not the same thing as an answer from a person who has that job, that frustration, and a Tuesday afternoon that went sideways because of a broken export.
Discovery exists to find the second kind of answer. The gap between the two is where products get built wrong.
What happens when you check it against real people?
Researchers have been checking, and the pattern is remarkably stable across disciplines.
Nielsen Norman Group took three of its own past studies that had used real participants and reran them with synthetic users from a dedicated platform and from ChatGPT. In a study on self-paced online learning, a real participant admitted finishing three of seven courses. The synthetic participants said they had completed every course they mentioned. Real learners mostly avoided discussion forums, calling them contrived; the synthetic learners said forums were important. When the team tested a drone delivery concept, a synthetic user called it “a game-changer.” The authors’ summary is blunt: synthetic users idealize behavior and want to please.
Survey researchers found the same thing at larger scale. A Vanderbilt team asked ChatGPT, prompted with personas, to reproduce answers from the American National Election Study. The published paper in Political Analysis found that the averages looked close to the real survey, but the synthetic responses had less variation, regression coefficients often differed significantly from the real data, small changes in prompt wording shifted the results, and the same prompt produced significantly different answers over a three-month window.
MeasuringU, a firm that does quantitative UX research, published a review of 12 peer-reviewed experiments in April 2026. Across those papers it counted 9 encouraging findings and 14 discouraging ones. One study that tried to replicate 14 classic social science experiments got 3 clean replications. Six of the 14 produced data with too little variability to analyze at all. The review’s recurring observation: correlations with human data are sometimes high, while the means, the spread, and the subgroup results drift.
That last point matters more than it sounds. A high correlation means the model ranks things in roughly the same order people do. It does not mean it gets the size of the effect right, and it says nothing about the people at the edges.
Why does it fail in exactly the places discovery cares about?
Look at what the failures have in common: less variance, more agreement, idealized behavior, flattened subgroups. Now look at what good discovery is supposed to surface: the outlier workflow, the user who hates the thing everyone else tolerates, the gap between what people say they do and what they actually do.
The model’s errors are concentrated on the exact signal you went looking for.
Work published in Nature Machine Intelligence by Angelina Wang, Jamie Morgenstern and John Dickerson tested LLMs prompted to take on demographic identities against real people holding those identities. Across every question type and group they tested, the model’s answers were less diverse than the humans’. The authors trace part of this to training data: text about a group is often written by people outside that group, so the model learns how outsiders describe someone, not how that person describes themselves.
For a product manager, translate “demographic identity” into “segment.” Your enterprise admins, your night shift users, your customers in a regulated industry, your users who churned in month two: the less those people write publicly about their work, the thinner the model’s picture of them. That is usually the segment you most needed to hear from, which is also why it was hard to recruit in the first place. I wrote earlier about how who you recruit decides what discovery tells you; synthetic users are the extreme case of that problem, because the “recruit” is an average of the internet.
Sycophancy compounds it. The same leading question that skews a human interview (the kind of question I covered in Your Customers Aren’t Lying. Your Questions Are.) skews a model far more, because agreeing with the person asking is close to what the model was tuned to do. A human might push back on a bad concept out of irritation. A synthetic user almost never will.
Is there anything it’s good for?
Yes, and the honest answer is more useful than a blanket ban.
The NN/g team found one case where synthetic output held up: asking a synthetic pharmaceutical sales rep to describe a typical workday produced something close to what real reps had said. That is desk research. Well-documented roles, common workflows, and broad attitudes on widely discussed topics are where the training data is dense, and the model does a reasonable job summarizing it.
There is also older work on pricing. A Harvard Business School working paper by James Brand, Ayelet Israeli and Donald Ngwe found GPT’s willingness-to-pay estimates for familiar consumer goods, like fluoride toothpaste, landed near figures from human studies. Note the conditions: a familiar category, a mature market, and plenty of published consumer data. Those are close to the opposite of the conditions under which a product team usually needs discovery.
So the defensible uses look like this:
- Rehearsing a discussion guide. Run your interview script against a synthetic participant to find confusing wording, questions that lead, and gaps in the flow before you spend a real person’s hour.
- Generating hypotheses to test. Ask a model what objections a procurement lead might raise. Treat the list as a starting set of assumptions, then go find out which ones are true.
- Getting oriented in an unfamiliar domain. Before your first call with hospital billing staff, a synthetic briefing on their day is better than walking in cold.
- Stress-testing your own reasoning. Ask the model to argue against your concept. It is better at playing critic when you explicitly ask than at volunteering criticism as a “user.”
Every one of those uses happens before real research or alongside it. None of them replaces it.
How do you keep it from quietly replacing the real thing?
This is the operational question, and it is the one I would worry about most as a leader. The research risk is real, but the bigger risk is organizational drift.
In network operations, every change I approved went through a lab environment first. The lab was genuinely valuable; it caught configuration mistakes cheaply. It also never once told us what production traffic would do at two in the morning, and the teams that started treating a clean lab run as proof of safety were the teams that caused outages. The tool was fine. The habit that grew around it was not.
Synthetic research invites the same habit. NN/g flagged this directly: teams that start with synthetic users can get comfortable and stop doing real research. A few guardrails help.
Label it everywhere. Any slide, doc, or ticket that cites synthetic output says so in plain words. A synthetic quote presented next to a customer quote, unlabeled, will be remembered as a customer quote by the time it reaches a planning meeting.
Never let it close a decision. Synthetic output can open a question or sharpen one. It cannot be the evidence that ends a debate about whether to build something. If the decision rests on it, you have not done the research yet. Scoping research to the decision, which I have argued for here, makes this rule easy to enforce: name the decision, then ask whether the evidence came from a person.
Track the ratio. If your team’s count of real customer conversations per month is falling while synthetic sessions rise, that trend is the problem, regardless of how good any individual synthetic session looked.
Check it against what you already know. Before trusting a synthetic persona for a segment, run it on questions where you already have real answers from that segment. If it confidently gets those wrong, you have learned how far to trust it on the questions you don’t have answers to.
Rerun it. Because prompt wording and model versions shift the output (the Vanderbilt team saw this over just three months), a finding you cannot reproduce next quarter was never a finding.
The appeal of synthetic users is that they remove the friction of talking to customers: the scheduling, the no-shows, the awkward silences, the person who wants to talk about something else entirely. Most of that friction is where the information lives. A participant who rambles off topic about a workaround you never knew existed is giving you the most valuable ten minutes of the quarter. A model will stay on script, agree with your framing, and hand you back the average of what you already suspected.