Most survey problems don't arise in the analysis module. They arise when the questions are written. Here are the seven most common mistakes in writing items — and a small checklist to help you catch them before the field phase.
A poorly written question produces poor data. That sounds obvious, but it's the underrated reality of survey research: respondents answer exactly what's written — not what the researchers actually wanted to know. When those two things diverge, the analysis is salvaged but the answer is worthless.
It pays to read through your items methodically before every field phase. What we've put together here is the order in which we do that in our own studies: first the framework of thinking, then the typical mistakes, and finally a short checklist.
Before you write: the six W-questions
Before you even write a single question, clarify for yourself the six fundamental W-questions about the survey. It seems formal, but it saves a lot of time later:
- Who is being surveyed? — This determines language, readability, and technical vocabulary.
- What exactly do you want to measure? — The construct, not the item. Employee satisfaction is the construct; "I feel appreciated" is one of several items measuring it.
- When does the survey take place? — Before or after an event, in which life phase, in which mood?
- Where is the answer given? — On a smartphone on the go, on a desktop in the office, in a phone interview? This affects the length and complexity of every question.
- Why are you asking the question? — What decision is supposed to be made based on the answers? If you can't answer that, the question is unnecessary.
- How will the answer be used later? — Mean comparison, frequency analysis, correlation? This determines the level of measurement.
Anyone who doesn't answer these six questions in advance writes items that turn out to be useless for later analysis. The most common weak spot is the fifth — in practice, "why" is often answered with "because it would be interesting." That's not enough.
Seven typical mistakes in writing questions
In our joint analyses with universities and mid-sized companies, we keep seeing the same seven types of mistakes. They're not random — they're the tripwires that catch everyone in the first draft. Knowing them helps you avoid them.
1. Double-barreled questions
A double-barreled question asks two things at once but allows only one answer. "Are you satisfied with the price and the service?" — someone who is satisfied with the price but dissatisfied with the service has no honest way to answer.
2. Leading questions
A leading question suggests the desired answer through word choice, order, or tone. Anyone who reads "Would you also agree that our new product represents a significant innovation?" already has the cue to the "correct" answer built into the sentence structure.
3. Implicit assumptions
A question with implicit assumptions presupposes knowledge or an action you haven't verified. "How often do you buy our product?" — the question assumes that respondents even know your product and have bought it before. Someone who hasn't doesn't know how to answer, and often picks "rarely" — which distorts the analysis.
4. Hypothetical statements
Questions of the type "If X happened, what would you do?" yield unreliable data. What people <em>say</em> they would do systematically differs from their actual behavior. Hypothetical statements are sometimes the only option in the exploratory phase of a new concept — but as a prediction of behavior they're methodologically weak.
5. Generalizations and vague frequency terms
"How often," "usually," "mostly" — frequency adverbs like these are interpreted very differently by different respondents. What one person considers "often" (once a week) is rare for another. This variability feeds directly into the data and makes comparisons unreliable.
6. Technical jargon and abbreviations
Phrases that you and your colleagues use every day are often incomprehensible to outsiders. "Rate your experience with the CX touchpoint using NPS logic" makes marketing pros happy and puts off 80% of respondents. Rule of thumb: write for someone reading about the topic for the first time.
7. Unbalanced answer options
When the answer scale contains more positive than negative options, responses systematically drift in the positive direction — even when that doesn't reflect the actual opinion. This applies not only to Likert scales, but to any answer list with an evaluative character. More on this in our detailed article Likert scales — five aspects.
Checklist before the field phase
Before you put a survey online, go through each question one by one — ideally together with someone from the target group who wasn't involved in developing it. These eight points as a filter:
- One question = one thing. No double-barreled constructions.
- Linguistically neutral. No "also," "of course," "significant" in the question.
- No implicit assumptions about respondents' knowledge or behavior.
- Ask about actual behavior wherever possible, rather than hypothetical behavior.
- Concrete time and quantity figures instead of vague frequencies.
- Everyday language, no jargon, no abbreviations without explanation.
- Symmetric answer scales with clear levels.
- A pre-test with at least three people from the target group before the field starts.
The pre-test is the underrated step in this process. Even two hours with three test participants usually uncovers problems that would otherwise have cost hours or days in the analysis.
Conclusion
Good questions don't come from talent, but from discipline. Anyone who answers the six W-questions in advance, knows the seven tripwires, and runs a short pre-test before the field has already avoided most of the later data problems. What remains — analysis, aggregation, interpretation — is then a different discipline.
If you're unsure whether a question holds up methodologically for a specific survey, write to us. We answer methods questions even without a contract.
Sources
- Porst, Rolf: Fragebogen — ein Arbeitsbuch. 4th edition. Springer VS, 2014.
- Mummendey, Hans Dieter, and Ina Grau: Die Fragebogen-Methode. 6th edition. Hogrefe, 2014.
- Schnell, Rainer, Paul B. Hill and Elke Esser: Methoden der empirischen Sozialforschung. 11th edition. De Gruyter Oldenbourg, 2018.