At Airbnb, about 8 percent of search experiments improved the metric they were built to move. At Booking.com, Google Ads and Netflix the figure sits around 10 percent. Bing runs at roughly 15 percent, and Microsoft as a whole, the best performer on the list, lands near 33 percent. Those numbers come from a 2022 KDD paper by Ron Kohavi, Alex Deng and Lukas Vermeer, three people who between them have run tens of thousands of controlled experiments at Airbnb, Booking, Amazon and Microsoft.
Read that list again with your own roadmap in mind. These are some of the most sophisticated product organizations on earth, staffed with people who study their users for a living, and most of what they ship as a test does nothing or makes things worse. If your team’s ideas are winning far more often than theirs, the likelier explanation is not that you are better at product. It is that you are not measuring.
The failure rate is the normal state of product work
Kohavi’s team at Microsoft first published the pattern in a 2009 internal paper on online experimentation. Their summary was blunt: “only about 1/3 of ideas improve the metrics they were designed to improve.” The rest split roughly evenly between flat and negative. These were not throwaway ideas. They had passed design review, been prioritized and been built.
The paper also flagged an inconvenient follow-on. When people first saw the statistics, many dismissed them, assuming their own group was different. Then the experiments came back and Microsoft turned out to be no different at all.
The 2017 Harvard Business Review article by Kohavi and Stefan Thomke put the same point in commercial terms. At Google and Bing, only about 10 to 20 percent of experiments generate positive results, and “even experts often misjudge which ones will pay off.” The same article tells the story of a small Bing idea to lengthen ad headlines. Program managers ranked it low among hundreds of others and it sat untested for more than six months. When an engineer finally ran it, revenue rose 12 percent, worth more than $100 million a year in the United States alone. The best idea in the backlog was the one the prioritization process had buried.
So the failure rate cuts both ways. Your confident bets mostly do not work, and your ignored ideas occasionally do. Intuition is bad at both ends.
Some of your winners are not winners either
This is the part most teams never hear. When the prior odds of an idea working are low, a statistically significant result is far less trustworthy than the p-value suggests.
The KDD paper calculates what it calls false positive risk: the chance that a “winning” test is actually a dud that got lucky. With a 33 percent success rate and a well-powered test, about 6 percent of declared wins are false. Drop the success rate to Airbnb’s 8 percent and that risk climbs to 26 percent. Run an underpowered test at 20 percent power, which is common on low-traffic products, and more than half of your significant results at a 10 percent success rate are false positives.
Think about what that means for a roadmap. A team celebrates three wins this quarter, builds the next quarter on top of them, and one or two of the foundations were noise. The paper’s recommendation is simple: treat surprising results with suspicion and replicate them before you bet on them. The authors invoke Twyman’s law, the old analyst’s rule that any figure that looks interesting or different is usually wrong.
What changes when you accept the odds
I learned a version of this in IT operations long before I saw anyone’s experiment data. Running operations at a large telecom, we pushed a lot of changes that everyone in the room agreed would reduce incidents or cut handle time. When we started doing honest post-implementation reviews, comparing the numbers before and after instead of asking the change owner how it went, a sobering share of those changes had done nothing measurable. The ones that worked were often not the ones we had argued hardest for. Nobody’s judgment got worse when we started measuring. We just stopped getting to grade our own homework.
For a product manager, accepting a 10 to 30 percent hit rate changes several habits.
Make ideas cheap before you make them big. If two out of three ideas will not move the metric, the cost of each test matters more than the brilliance of any single idea. The Bing headline change took days of engineering. A cheap fake door test or a rough prototype lets you take ten swings for the cost of one polished build.
Write down the expected result before the test runs. Name the metric, the direction and the size of change you expect. This is the only reliable way to stop a team from finding something that went up afterward and calling it the goal all along.
Stop treating seniority as a forecast. The original Microsoft paper framed controlled experiments as a way to settle debates with data instead of deferring to the highest-paid person’s opinion. If even experts misjudge which ideas will pay off, the loudest voice going first is no better a predictor than anyone else in the room.
Replicate anything that looks too good. A 30 percent lift on a mature funnel is far more likely to be a bug, a tracking change or a lucky sample than a breakthrough. Rerun it. Even Bing’s headline result first tripped an alert designed to catch numbers too good to be true, and the team treated it as a probable bug until the checks came back clean.
Report the losses. A team that only presents wins teaches leadership to expect a 100 percent hit rate, which guarantees that the next honest quarter looks like a failure. Kohavi’s own Airbnb tenure is a useful counterexample: the relevance work he led delivered more than 6 percent improvement in booking conversion, worth hundreds of millions of dollars a year, while the vast majority of individual experiments failed. The value came from the portfolio, not from any one idea being right.
The point of discovery was never to be right about your ideas. It was to find out quickly and cheaply which ones are wrong, so the rare one that works gets the resources it deserves.