Home - Product Management Resources

In 1936, Literary Digest ran the largest election poll anyone had ever attempted. The magazine mailed out ten million straw ballots, got back about 2.3 million, and on the strength of that enormous pile confidently predicted Alf Landon would beat Franklin Roosevelt in a landslide. Roosevelt won 46 of 48 states. The poll failed spectacularly not because the sample was too small but because of who was in it: the mailing list was built from automobile registrations, telephone directories, and the magazine's own subscribers, which in the middle of the Depression skewed heavily toward people wealthy enough to be unrepresentative of the electorate. The sample was gigantic. The sample was also wrong. Product discovery fails the same way, for the same reason. Teams obsess over interview technique: the questions, the phrasing, whether they accidentally led the witness. They spend almost no structured thought on the decision that determines the answer before a single question gets asked, which is who ends up in the room. Recruiting is the research. Everything downstream inherits whatever bias you baked into the guest list. The people easiest to reach are the least representative Recruiting for convenience is the default, and it has a predictable shape. You talk to your most active users, the accounts your customer success team happens to like, the people who reply to an email within the hour, the ones who surface in your in-app survey because they were engaged enough to be using the product when the prompt popped. In research terms this is a convenience sample: data collected from whoever is accessible rather than whoever is representative. It is fast, it is cheap, and it quietly poisons the well. The distortion runs in one direction. Convenience samples over-represent the motivated and the satisfied. Your happiest, most engaged users tell you the product is great and ask for more of what they already use. The people who would have told you something uncomfortable, the ones who bounced in the first week, who evaluated you and picked a competitor, or who never engaged enough to answer a survey at all, are structurally absent from the conversation. You are not hearing a market. You are hearing your fan club, amplified. The recruiting channel itself introduces bias before anyone speaks. Nielsen Norman Group's guidance on recruiting and screening research participants catalogs the usual failure modes: professional testers on panels who exaggerate feedback to match what they think you want, internal user panels whose brand loyalty produces praise instead of realistic insight, coworkers who are reluctant to say anything genuinely negative, and online communities where a few vocal members create the impression of consensus. None of these people are lying. They are simply not the people whose behavior you are trying to predict. Who is missing matters more than who showed up The Literary Digest lesson restated for product work: the danger is not the 2.3 million people who mailed a ballot back, it is the systematic gap between who responded and who didn't. In discovery the silent segment is usually the whole ballgame. The customer who churned without a word is telling you something more valuable than the power user who has an opinion about every button. The prospect who looked at your pricing page and left is a better source of truth about your positioning than the champion who already signed. This is why so much research produces confident conclusions that the market later contradicts. A roadmap built entirely on the input of current, active, willing customers will optimize hard for the people you already have and stay blind to everyone you are failing to reach. It is a close cousin of the mistake I wrote about in why a real problem still does not guarantee a real market: the evidence looks solid because the sample was rigged, gently and unintentionally, toward agreement. The same trap sits inside exit data, which is why the exit survey is the least honest data most teams own: the people motivated enough to fill it out are rarely the people who left for the reasons that matter most. Academic research has the same blind spot at scale. A large share of published psychology rests on samples of Western, educated undergraduates who happened to be available in a lecture hall, a population so unrepresentative that researchers now flag it explicitly as an outlier rather than a neutral default. If a whole discipline can fool itself for decades by talking to whoever was standing nearby, a product team running six interviews a quarter should assume it can too. Fixing the guest list before you fix the questions Erika Hall put the priority in the right order in Just Enough Research. As she frames it, "Ask the right people the wrong things, and you'll still learn something. The wrong people won't help you." Her analogy for recruiting is fishing: decide what kind of fish you want, make a net, go to where the fish are, and collect the ones you actually need. The point is that recruiting is a deliberate act of targeting, not a scramble to fill slots with whoever answers. Three moves separate deliberate recruiting from convenient recruiting. Screen by recent behavior, not by availability. Teresa Torres builds her continuous discovery method on exactly this: recruit customers based on a trigger, someone who just completed the specific action you care about, and automate that recruiting so you are not hustling to find a warm body every week. Behavior is a far better filter than willingness. The person who did the thing yesterday remembers it. The person who volunteered because they love talking to you is selecting themselves for reasons that have nothing to do with your research question. Deliberately recruit the segment that will disagree with you. Set an explicit quota for churned users, evaluators who chose someone else, and people who abandoned before activating. These conversations are harder to schedule, which is precisely why convenience recruiting skips them, and precisely why they carry the most information. If every person you interviewed likes you, you funded a testimonial, not a study. Write down who you did not reach. NNG recommends screening interviews and reviewing the responses of disqualified candidates, not just the ones who made the cut. The discipline that matters is naming the gap out loud: which segments are absent from this round, and what would change if they were present. A finding you cannot generalize is not automatically useless, but pretending it generalizes when your sample cannot support it is how teams talk themselves into building the wrong thing. The part that costs real money Running operations at a large telecom years ago, I watched a product group spend a quarter interviewing the enterprise accounts that already loved the platform, then present a roadmap that leaned hard into deepening features those accounts requested. The research was clean. The interviews were well run. The problem was that every account in the sample was a customer who had already chosen us and stayed, so the roadmap doubled down on the segment least at risk of leaving and learned nothing about the mid-market deals we kept losing. The losses were not in the data because the losers were never in the room. Fixing that did not require better interview questions. It required a different invite list. Interview technique is the last ten percent of discovery quality. Who you invited is the first ninety. Before your next round, audit the guest list before you polish the discussion guide, and ask the uncomfortable question the Literary Digest editors never did: is this the population I need to hear from, or just the population that was easy to reach?

Recent Posts