External email survey response rates now average between 15% and 25%, down roughly 1 to 2 percentage points per year since 2019. Those numbers alone should give any product manager pause before creating another Google Form. But the real problem with product surveys runs deeper than low participation: most surveys ask questions that cannot produce actionable discovery insights.
The Comfort of Numbers
Surveys appeal to product managers for obvious reasons. They scale. They produce quantitative data. They feel democratic (“we asked 500 users!”). And they require less emotional labor than sitting across from a customer for 45 minutes of conversation.
That combination makes surveys the default when a PM feels pressure to validate a direction. The instinct is understandable. The results are usually disappointing.
The issue is structural. Surveys capture stated preferences, and stated preferences are unreliable predictors of behavior. Rob Fitzpatrick articulated this problem in The Mom Test: people are overly optimistic about what they might do in the future. When you ask “Would you use this feature?” the honest answer is almost always “probably,” which tells you nothing actionable.
Erika Hall, author of Just Enough Research, went further. She left surveys out of the first edition of her book entirely because she considers them “the most dangerous research tool,” one that straddles qualitative and quantitative research and, at its worst, represents the worst of both.
Three Ways Surveys Mislead Discovery
They measure opinion, not behavior. A survey asking “How important is reporting to you?” will almost always return high scores because nobody wants to say reporting is unimportant. But pull the usage logs and you’ll often find that fewer than 12% of users opened a report in the past 30 days. The gap between what people say matters and what they actually do is the central problem of survey-based product discovery. Behavioral data and direct observation reveal what surveys cannot.
They attract the extremes. The people who respond to surveys tend to be either deeply satisfied or deeply frustrated. The silent majority in the middle opts out, creating non-response bias that skews every conclusion. A 2025 analysis from the San Francisco Federal Reserve found that declining response rates are now threatening the reliability of even government economic surveys. Product surveys, with far less institutional authority behind them, face an even steeper version of this problem.
They confuse research questions with survey questions. This is the mistake Erika Hall identifies as the root cause of bad survey research. A research question might be “How do our enterprise customers evaluate ROI before renewing?” That is a legitimate discovery question. But you cannot ask it directly on a survey and expect a useful answer. Translating research questions into valid survey instruments requires statistical training that most product managers do not have, and skipping that translation step produces data that looks rigorous but is not.
When Surveys Actually Earn Their Place
None of this means surveys are useless. They work well in specific, bounded situations.
Prioritization among known options. If you have already identified five potential improvements through customer interviews and observation, a survey can help you understand which ones matter most across the broader user base. The key is that the options come from qualitative discovery first. The survey validates relative priority; it does not generate the options.
Satisfaction tracking over time. NPS and CSAT scores are noisy in isolation, but tracked quarterly across the same cohort, they reveal directional trends. The value is in the trend line, not any single score.
Segmentation and demographics. Surveys are efficient for collecting firmographic data: company size, role, industry, usage frequency. This is factual, not opinion-based, which sidesteps the stated-preference problem entirely.
Post-experience feedback. A three-question survey triggered immediately after a user completes a specific workflow can capture in-context reactions while the experience is fresh. Teresa Torres, author of Continuous Discovery Habits, recommends weekly customer touchpoints; a brief contextual survey can supplement (never replace) those conversations.
The Question Before the Survey
Before creating any survey, ask one question: “What will we do differently depending on the answers?”
If the answer to every possible survey result is the same course of action, the survey is a waste of your users’ time and your own. If a result of 4.2 out of 5 leads to the same decision as 3.8 out of 5, the survey is theater.
On a fractional COO engagement in 2024, I watched a B2B SaaS product team spend two weeks designing a 40-question survey, wait another three weeks for responses, and then realize the results did not change a single roadmap decision. The survey existed to create the appearance of rigor. The actual discovery work (three targeted customer conversations) could have happened in a single afternoon and would have surfaced the real blockers.
What to Do Instead
For most discovery questions, direct conversation with users produces more reliable signal in less time. Five well-structured interviews will reveal patterns that a 200-response survey cannot surface, because interviews let you follow the thread when something unexpected appears. Product analytics reveal what users actually do, which is more trustworthy than what they say they would do. And prototype testing lets you observe real behavior before committing engineering resources.
Surveys have a place in the product manager’s toolkit, but it is a narrow one. The next time you feel the pull to “just send out a quick survey,” pause and ask whether the question you need answered is actually a survey question, or whether it is a conversation waiting to happen.