Two questionnaires with identical content can lead to different results — purely because of the order of the questions. Order effects are well documented in survey research and affect both dropout rates and the validity of the answers. Which effects exist, and how to counter them.
You've built a carefully worded questionnaire — and you're getting data that doesn't match your expectations. Before you start doubting your wording or your target group, it's worth taking a look at the order of your questions. Order effects are among the best-documented and most frequently underestimated sources of systematic bias in online surveys.
Research distinguishes two families of order effects. One raises the dropout rate: respondents leave the survey before they've completed it. The other distorts the answers: respondents stay, but give different answers than they would have with a different arrangement. Both families are avoidable — if you know what to look for.
Why order distorts in measurable ways
The order of the questions is not neutral. Every question creates a cognitive context that influences the questions that follow. A question about stress at work, followed by a question about general life satisfaction, systematically yields lower satisfaction scores than the reverse order. These biases are not small: shifts of 10 to 30 percentage points are documented in the methodological literature.
The same applies to the dropout rate. An opening question that is too difficult or too personal means that a considerable share of respondents never even begins. Anyone who sees only gaps here misses the real problem: the missing answers aren't randomly distributed but systematic — and thus a methodological risk to the representativeness of the entire sample.
Dropout effects: why respondents leave early
Dropout effects arise when the arrangement of the questions overwhelms respondents emotionally, cognitively, or in terms of time. Four patterns come up especially often in practice.
The opening question
The first question decides the success or failure of a survey disproportionately often. Within the first 30 seconds, the respondent decides whether they take the survey seriously, whether the questions appeal to them, and whether they consider the effort worthwhile. If the opening question is too personal, too difficult, or simply uninteresting, the dropout rate jumps sharply.
The wrong order of information types
In most surveys, three kinds of information are collected. Substantive information — the actual questions the survey is about. Classifying information — demographic characteristics such as age, gender, education. Identifying information — contact details, in case the survey is anonymous with optional follow-up.
The optimal order is: substantive → classifying → identifying. The reason is simple: engage respondents on substance first, let them feel that their opinion matters — then they'll be willing to add a few demographic details at the end. Start instead with age, gender, and postal code, and you signal: "This isn't about you, this is about sorting you" — and you lose exactly the respondents you actually wanted to win over.
From practice: whoever starts with demographics has almost always already lost. The dropout rate in the first three questions is the critical metric of any online survey — and demographics at the start are one of the most reliable dropout drivers there is.
So: demographics at the end. In the worst case you lose the classification — but you've already gathered the substantive core in the first questions. In the best case you get both, because by then respondents are far enough into the survey that the barrier to dropping out is higher than the barrier to filling in the demographics. A decision with no downside scenario — rare enough.
Difficult questions at the start
Complex, cognitively demanding, or emotionally unpleasant questions don't belong at the start. A six-point scale with a double negative, a question about income or political attitudes, an open question with a high expectation of the length of the answer — all of these work considerably better in the middle section of the survey. There, respondents are already engaged and the cognitive willingness to make an effort is higher.
Illogical arrangement
When, from the respondents' perspective, the questions follow no comprehensible thread, participation drops. Jumps between topics, abrupt shifts in tone, or obvious logic errors — for example a brand question after a respondent has denied the preceding ownership question — create irritation. Irritation is a reliable harbinger of dropout.
Bias effects: why answers deviate systematically
Bias effects are subtler than dropout effects — and in many respects more problematic. The survey gets completed, the data looks complete, but it doesn't measure what it's supposed to measure. Three mechanisms are particularly well established in online survey research.
Primacy and recency effects
With multiple choice questions, the first and last answer options are chosen disproportionately often. The effect is well documented and appears in two variants: with visual presentation (online, paper), the primacy effect dominates — the first options are preferred. With auditory presentation (telephone interviews), the recency effect dominates — the last options are preferred because they are still in short-term memory.
The bias can be substantial. Studies show that in online surveys the first answer option is chosen 5 to 15 percentage points more often than corresponds to its actual position in the preference distribution. With scale questions that have many response levels, the effect can shift entire distributions.
Halo effect between questions
A preceding question colors the one that follows. Someone asked first about the risks of a technology and then about their assessment of that technology answers, on average, more skeptically than someone shown the reverse order. The mechanism isn't manipulation in the narrow sense — it's an entirely natural consequence of human information processing: recently activated content is more strongly present in working memory and influences subsequent judgments.
A classic example from the research: the question "How satisfied are you with your life?" systematically yields lower scores when it comes after a question about work-related stress, and higher scores when it comes after a question about close friendships. Schwarz and Strack (1991) quantified the effect experimentally: the shift can amount to one or two scale points on a seven-point scale.
Biased questions
Related to the halo effect, but located within a single question: a question is worded so that an argument or a judgment is already built in. "Should speed limits be introduced, even though they restrict individual freedom?" yields different agreement scores than "Should speed limits be introduced to reduce traffic fatalities?" The substantive core of the question is identical — the framing is not.
How to avoid order effects
Order effects can't be fully eliminated — they are a property of human information processing. But they can be reduced considerably with eight field-tested measures.
Eight measures for clean questionnaire architecture
- A simple, neutral opening question. A question that is quick to answer, embarrasses no one, and directs attention to the topic. Rule of thumb: if the question takes longer than ten seconds, it's too hard for position one.
- Information types in the right order. Substantive questions first, then demographic ones, and last — if at all — identifying details. Start with the postal code, and not infrequently you never even capture any answers.
- Difficult questions in the middle section or at the end. Complex scales, sensitive topics, open answer fields with long responses — anything that means effort — doesn't belong at the start.
- A clear thread and topic grouping. Questions on the same topic area stay together. Transitions between topics are marked with a short note. Jumps without preparation create irritation.
- Use display and skip logic. Someone who has indicated they don't own a smartphone shouldn't see the follow-up question about smartphone brand. QUESTIONSTAR provides this logic by default — without it, illogical arrangement becomes a cause of dropout.
- Randomize the order of the answers. With multiple choice questions, the answer options are shown to each respondent in random order. This spreads the primacy effect evenly across all answer options and cancels it out in aggregation.
- Physical distance between halo-relevant questions. If two questions could influence each other, place other questions in between. Not elegant, but effective: the further apart two questions are, the less the first affects the second.
- Funnel approach: from the general to the specific. General attitude questions first, concrete detail questions after. This keeps a specific question from steering the general assessment in a particular direction.
Conclusion
Order effects are not a marginal phenomenon. They operate in nearly every online survey and can considerably undermine the validity of the results. The good news: they can be controlled with manageable effort. Anyone who consistently applies the eight measures above has most of the typical biases under control.
For methodologically especially sensitive studies — for instance if you want to publish with the data or make strategic decisions — an additional step is advisable: a split-half design. You create two variants of your questionnaire with different question orders, randomize the assignment of respondents, and compare the results. If the difference is substantial, you've identified an order effect. If it's small, you can publish with greater confidence.
Sources
- Malhotra, Naresh K., and David F. Birks: Marketing Research. An Applied Approach. 4th edition. Pearson Education, 2017.
- Schumann, Siegfried: Repräsentative Umfrage. Praxisorientierte Einführung in empirische Methoden und statistische Analyseverfahren. 7th edition. De Gruyter Oldenbourg, 2019.
- Schwarz, Norbert, and Fritz Strack: Context Effects in Attitude Surveys: Applying Cognitive Theory to Social Research. European Review of Social Psychology, 2 (1), 31—50, 1991.
- Bogner, Kathrin, and Uta Landrock: Antworttendenzen in standardisierten Umfragen. GESIS Survey Guidelines. Mannheim: GESIS Leibniz-Institut für Sozialwissenschaften, 2015.