research methods
Sample size and statistical significance
Why does sample size matter, and what does statistical significance mean?
Sample size is how many people (or sessions) a finding rests on; statistical significance assesses how surprising your measured difference would be if there were really no effect, under a specified model and assumptions. It does not prove an effect is real, important or repeatable. Small samples swing wildly by chance, so a difference can look dramatic and still mean little - which is why significance goes hand in hand with sample size and effect size.
Also known as: sample size, statistical significance, significance, sampling
Prefer to watch?
Watch the recap 0:56
The demo
Every coin here is perfectly fair - exactly 50/50. Pick a sample size, flip, and watch how far the result strays from the truth. Then do it again, and again.
5 · 10 · 30 · 100 · 300 · 1000
Press “Flip the coins”.
What this demo shows (text version)
A simulator flips a chosen number of perfectly fair coins and shows the observed percentage of heads against the true 50% line. With a small sample (say 5 or 10), the result swings wildly - 70%, 30%, 80% - and looks like a biased coin, even though every coin is fair. As you raise the sample size to hundreds, the result settles close to 50%. Running it repeatedly makes the volatility of small samples obvious.
The lesson maps straight onto research: small samples are noisy, so a difference between two designs measured on few users or conversions can be large by pure chance and mean nothing. Statistical significance estimates whether a gap could be random; quantitative claims need enough sample to pass it. (Qualitative usability testing is different - a few users reliably surface most problems, because you're finding issues, not estimating a number.)
Small samples lie, convincingly. A handful of coin flips can land 70% heads, and a handful of users can make a worse design "win" by chance. Significance asks how likely a gap this size would be if nothing were going on - it flags noise, it doesn't prove an effect is real, meaningful or replicable. Pair it with effect size and judgement: quantitative claims ("B converts better") need enough sample; qualitative insight (five users finding problems) is a different game with different rules.
Flip just a few coins and you'd swear the coin was biased - 70%, 30%, all over the place. Flip hundreds and it settles near the true 50%. Nothing about the coin changed; only the sample size. That's why a snap A/B result on a few hundred visits can reverse itself later: the early "winner" was noise wearing the costume of signal.
The core intuition is that small samples are noisy. A fair coin flipped ten times can easily land seven heads; flipped a thousand times it hugs 50%. The same volatility means an A/B test with few conversions can show a big "lift" that's pure chance - which is why teams who peek early and ship the leader so often see the effect evaporate.
Statistical significance (often a p-value, with 0.05 a common-but-arbitrary threshold) estimates how likely a result this extreme would be if there were really no difference. It is not "how big" or "how important" the effect is, and it can be reached on a trivial difference with a huge sample, or missed on a real one with too small a sample. Pair it with effect size and confidence intervals.
Crucially, the rules differ by method. Quantitative claims - conversion rates, metric differences - need enough sample and proper significance testing before you trust them. Qualitative usability research is the opposite: a handful of users reliably surfaces most usability problems, because you're looking for issues to fix, not estimating a population number. Match the sample logic to the question you're actually asking.