metrics
Task success rate
What is task success rate?
Task success rate is the share of people who finish a task - four of five users complete it, so 80%. It is the most basic measure of whether a thing works at all, and the easiest to read too much into.
Also known as: completion rate, task completion rate
The demo
Each chip is one user attempting the task. Click a chip to flip it between pass and fail, and watch the success rate. Then switch the sample size and flip again - notice how much less one user matters when there are more of them.
…
Click a user to flip pass / fail:
Flip a user and watch the rate move.
What this demo shows (text version)
A set of user attempts, each marked pass or fail, with a live success rate shown as a percentage and a count (for example, "80% - 4 of 5 users"). Clicking any user flips its outcome and recalculates the rate.
A toggle switches the sample between 5 users and 20 users. At 5 users, one flip changes the rate by twenty percentage points; at 20 users, the same flip changes it by five. The demonstration is that a success rate from a small sample is volatile - a single person swings it dramatically - so the percentage should always be read alongside how many people it is based on.
With five users, one person flipping from fail to pass swung the score twenty points. With twenty, the same flip barely moved it. The number didn't get more honest when it got more stable - it just stopped lying so loudly. A success rate without a sense of its sample size is a confident-looking guess.
The trap is precision the data can't support. "80% success" sounds exact, but at n=5 each person is worth twenty percentage points, so your true rate could sit anywhere across a wide band. Report the count alongside the percentage (4 of 5, not just 80%) so nobody mistakes a small sample for a settled fact.
Binary pass/fail also hides the texture. Someone who limped to the end after three wrong turns counts the same as someone who breezed through; a person who gave up one step from done counts as a total failure. Pair the rate with time on task and a watch of where people struggled, or you will optimise a number while the experience stays broken.