research methods
Survivorship bias
What is survivorship bias in UX research?
Survivorship bias is drawing conclusions from only the things that made it through, while the ones that didn't - and the lessons they hold - are invisible. In UX it's the trap of studying only the users who stayed: your analytics, reviews and surveys are full of survivors, and silent about everyone who hit a wall and left.
Also known as: survivorship bias, survivor bias, survival bias
Prefer to watch?
Watch the recap 0:50
The demo
Returning warplanes came back peppered with bullet holes. Where would you add armour? Decide from the damage you can see - then reveal what the data was hiding.
What this demo shows (text version)
A plane diagram shows red dots where returning warplanes were found to be hit. The intuitive conclusion is to add armour where the damage clusters. Revealing "what's missing" highlights the opposite areas - the engines and cockpit - which show no damage on the survivors. That's because planes hit there didn't make it back to be counted, so armour actually belonged in the undamaged-looking spots.
That's survivorship bias, and UX data is full of it: analytics, reviews and surveys capture the users who stuck around, not the ones who hit a problem and left. "Users love this" can simply mean everyone who didn't already churned out of the dataset. The fix is to seek the missing cases - interview churned users, study first-time failures, track drop-off - and for every metric ask who isn't in it.
Your data is mostly survivors. Analytics, app-store reviews and surveys capture the people who got far enough to be counted - not the ones who bounced, churned or never signed up. So "users love feature X" may just mean the people who hated it already left. The fix is to go looking for the missing data: talk to churned and lapsed users, watch first-time failures, and ask who isn't in your numbers and why.
Like the wartime planes, the obvious move - reinforce where you see the damage - is exactly wrong: the planes hit elsewhere never came back to be counted. Your product is the same. The bullet holes you can see are the survivors' complaints; the fatal ones are in the users who already left and aren't in your data at all.
The classic story: analysts studied returning warplanes to decide where to add armour, and were about to reinforce the most bullet-riddled areas - until someone realised the planes hit in the untouched areas were the ones that didn't return. Armour belonged where the survivors had no holes. The missing cases held the real information.
UX data is riddled with the same trap. Analytics only count people who reached the page; reviews skew to the delighted and the furious who stuck around to write them; surveys reach current users, not the ones who quietly left. "Engaged users do X" can simply mean everyone who didn't do X has already churned out of the dataset.
The cure is to hunt the survivors' opposite. Interview churned and lapsed users, run first-time and failed-task sessions, track where people drop out (not just where they succeed), and for every cheerful metric ask: who isn't represented here, and would they say the same? The most important evidence is usually the evidence that didn't make it into your data.