Watch
Sample size and statistical significance 0:56
A 56-second explainer of Sample size and statistical significance, narrated over the same demo you can try yourself.
Try the interactive Sample size and statistical significance demo
What this video covers
Sample size is how many people (or sessions) your finding rests on; statistical significance is the check on whether a difference you measured is likely real or just noise. Small samples swing wildly by chance, so a difference between two designs can look dramatic and mean nothing - significance testing is how you tell signal from luck.
Flip just a few coins and you'd swear the coin was biased - 70%, 30%, all over the place. Flip hundreds and it settles near the true 50%. Nothing about the coin changed; only the sample size. That's why a snap A/B result on a few hundred visits can reverse itself later: the early "winner" was noise wearing the costume of signal.
