Watch
A/B testing 0:39
A 39-second explainer of A/B testing, narrated over the same demo you can try yourself.
Try the interactive A/B testing demo
What this video covers
A/B testing shows two versions to two random groups and compares a metric - clicks, sign-ups, sales - to estimate which performs better. It measures the effect of those particular variants on that particular metric, for the people and period you sampled. It won't tell you about comprehension, accessibility, long-term or novelty effects, or unmeasured harms - and done hastily it manufactures false winners from noise.
Early on, one version looked like a clear winner - and if you'd called it then, you'd have shipped a difference that was pure chance. Pour in more visitors and the gap melts into the noise, because the two were really the same all along. That gap between an early lead and a real result is the whole discipline of A/B testing.
