research methods
Surveys and questionnaires
What makes a survey question good or bad?
Surveys and questionnaires collect structured answers from many people at once - cheap, scalable, quantifiable. Their weakness is that the data is only ever as good as the questions: a leading, loaded, double-barrelled or vague question quietly manufactures the answer, so a badly-written survey produces confident numbers that mean nothing.
Also known as: surveys, questionnaires, survey design, survey questions
Prefer to watch?
Watch the recap 0:51
The demo
Read this customer survey as if you were filling it in. Then hit “annotate” - and watch almost every question turn out to be quietly rigged.
What this demo shows (text version)
A short customer survey is shown with several deliberately flawed questions. An "annotate" toggle reveals the problem with each: a leading question that signals the answer it wants ("how would you rate our excellent new checkout?"), a double-barrelled one asking two things at once ("was checkout fast and easy?"), a loaded question that assumes something ("what did you love most about it?"), and a vague, unbalanced scale with no "not applicable" option.
The point is that a survey's data is only as good as its questions - flawed wording manufactures the answers and produces authoritative-looking but meaningless numbers. The fixes are to write neutral, specific, single-idea questions, offer honest options, keep it short, and pilot it on real people. And because surveys capture what people say rather than do, pair them with behavioural evidence.
A survey is only as honest as its questions. The common faults - leading ("how great was…?"), double-barrelled (two questions in one), loaded (smuggled assumptions), vague scales, and missing options - all bend the answers before anyone replies. Write neutral, specific, single-idea questions; pilot the survey on a few people; and remember surveys capture what people say, not what they do, so pair them with behaviour.
Annotate the survey and almost every question lights up with a flaw: it tells you the answer it wants, asks two things at once, or smuggles in an assumption. The numbers such a survey produces would look authoritative and mean nothing. Good question-writing is most of good survey research - and most of it is invisible until you go looking.
The recurring faults are nameable. Leading questions point at the desired answer ("how helpful was our excellent support?"). Double-barrelled ones ask two things at once ("was it fast and easy?") so a single answer can't address both. Loaded questions bury an assumption ("what did you like most?" presumes you liked something). Vague or unbalanced scales, and missing "not applicable"/"don't know" options, force false precision.
Beyond wording, structure matters: question order can prime later answers, fatigue degrades responses near the end, and required questions with no honest option push people to lie or quit. Keep surveys short, put sensitive or open questions thoughtfully, and always offer an out where a real answer might not fit your choices.
Two habits prevent most disasters: pilot the survey on a handful of real people and watch where they hesitate or misread, and remember that surveys measure stated attitudes, not behaviour - what people say they'd do diverges from what they do. Triangulate with analytics and observation, and treat clean-looking survey numbers from messy questions as the illusion they are.