Watch
Sample size and statistical significance 0:56
A 56-second explainer of Sample size and statistical significance, narrated over the same demo you can try yourself.
Try the interactive Sample size and statistical significance demo
What this video covers
Sample size is how many people (or sessions) a finding rests on; statistical significance assesses how surprising your measured difference would be if there were really no effect, under a specified model and assumptions. It does not prove an effect is real, important or repeatable. Small samples swing wildly by chance, so a difference can look dramatic and still mean little - which is why significance goes hand in hand with sample size and effect size.
Flip just a few coins and you'd swear the coin was biased - 70%, 30%, all over the place. Flip hundreds and it settles near the true 50%. Nothing about the coin changed; only the sample size. That's why a snap A/B result on a few hundred visits can reverse itself later: the early "winner" was noise wearing the costume of signal.
