Questionnaire pretest: how to test your survey before launch

How to pretest a questionnaire: choose methods, decide how many people to involve, spot problems and document changes. Includes a checklist.

In short: a pretest is a trial run of your questionnaire with a few people from your target group before the actual survey begins. It shows whether people understand the questions the way you mean them, whether answer options are missing, how long it takes to fill in, and whether skip logic and layout work on the devices you tested (Möhring and Schlütz 2019, pp. 182-184). For a first comprehension test, five to ten people are often enough (Jacob et al. 2019, p. 214). For a test under real conditions, it's ten to 50, depending on the study (Möhring and Schlütz 2019, p. 184).

In this article, I'll answer five questions, one after the other:

• Why does even a carefully written questionnaire need a pretest?
• What methods are there, and how do you run cognitive interviews?
• How do you check the technical setup of an online questionnaire?
• How many test participants do you need, and what should you watch for?
• How do you document the results and decide on changes?

At the end, you'll find a compact pretest plan as a checklist.

A slide from my course repeats the same word three times: "Pretest! Pretest! Pretest!" (Marx 2026, slide 115). You always read your own questionnaire the way you meant it. Your test participants read it the way it's written, and a pretest makes exactly that difference visible. How to build the questionnaire is covered in Questionnaire design. This article is about the last step before launch: the trial run.

Why does every questionnaire need a pretest?

A paper boat looks seaworthy on your desk. You only find out whether it floats once it's in the water. Questionnaires are no different. Schnell gives the underlying reason: there's no empirically proven theory from which you could derive the perfect questionnaire at your desk. That's why every questionnaire and every question has to be tested empirically before data collection, and that test is called a pretest (Schnell 2019, p. 123).

Clear questions that people understand differently

Let's look at an example. "Do you own a car?" sounds clear. Willis shows that this question can be read in at least three ways: does a pickup truck or an SUV count as a car? Does it mean you personally or your whole household? And do you own a car you lease? (Willis 2005, cited in Schnell 2019, p. 125). In a conversation, people can clear up this kind of ambiguity by asking. When someone fills in an online questionnaire alone, they usually can't.

Belson had respondents explain, after answering, how they had understood a question. Only 3 out of 52 understood the question about hours of TV on a "weekday" exactly as intended, twelve counted all seven days, and across all the questions tested, an average of 29% of respondents understood the questions as intended (Belson 1981, cited in Schnell 2019, p. 137).

Even professionally built questionnaires aren't free of this. In a pretest of an almost finished questionnaire, Kreiselmaier et al. found 279 problems in 1,741 question-and-answer sequences, that is, in 16% (Kreiselmaier et al. 1989, cited in Schnell 2019, p. 127).

Don't just ask "Was everything clear?"

The obvious idea is to send the link to a few colleagues and ask whether everything made sense. That doesn't help much. Respondents almost always try to give an answer, because they assume every question has a purpose (Schnell 2019, p. 127). In a test with a deliberately flawed questionnaire, respondents mainly noticed missing answer options, and about a third managed that. Only 3 to 5% noticed loaded or ambiguous questions (Hunt et al. 1982, cited in Schnell 2019, p. 126).

So a pretest needs a method: follow-up questions, observation and a record. Here's what you check:

Proven questions belong in the test too. You'll find out why in the FAQ.

What pretest methods are there?

Strictly speaking, there's no such thing as "the" pretest. Jacob et al. describe a series of tests run one after the other, known as multi-method pretesting (Jacob et al. 2019, p. 212). Schnell recommends at least two rounds: a developmental pretest for the first draft and a final pretest for the last corrections. After the final pretest, no new questions should be added (Schnell 2019, pp. 127-128).

The methods can be grouped by where they take place: at your desk without respondents, in the lab with individual test participants, or in the field under real conditions (Möhring and Schlütz 2019, p. 183). Online, a fourth step is added: the technical test. No method finds everything. A mix works best, and a critical desk review and a test under real conditions should always be part of it (Möhring and Schlütz 2019, p. 183).

At your desk: expert review and questionnaire conference

It's hard to spot your own mistakes, so get help before the first test run (Möhring and Schlütz 2019, p. 183). In a questionnaire conference, a small team goes through the draft question by question. In an expert review, you give it to people who have built surveys themselves and ask for comments (Jacob et al. 2019, p. 212). Schnell recommends two to three hours of discussion with colleagues who don't work on the project (Schnell 2019, p. 130).

Checklists help keep the review from stopping at gut feeling. Willis's Question Appraisal System asks for each question, among other things: are there unclear terms? Is a reference period missing? Does the question contain a false assumption about the respondents? Does it require respondents to perform a calculation or recall information that most people would struggle to provide? Do the answer options fit the question, are they mutually exclusive, and is there a suitable option for every possible answer? (Willis 2005, cited in Schnell 2019, pp. 132-133). The questionnaire appraisal system by Faulbaum et al. also checks whether scales are balanced and equally spaced (Faulbaum et al. 2009, cited in Möhring and Schlütz 2019, p. 184).

An AI assistant can work through a checklist like this, too. If QUESTIONSTAR is connected to ChatGPT or Claude through the MCP interface, the assistant gets the QUESTIONSTAR checklist and can go through your draft with it: double-barreled and leading questions, hidden assumptions, unbalanced scales, overlapping categories. It points out potential problems in your draft, but the decisions are yours.

Every checklist has a limitation: it rarely identifies problems that arise only from the sequence of questions, such as context effects or incorrect skip logic (Schnell 2019, p. 131). For that, you need people who actually fill in the questionnaire. How question order shifts answers is covered in Order effects in surveys.

In the lab: cognitive interviews

"Lab" sounds more elaborate than it is. It just means a quiet room for an uninterrupted conversation (Jacob et al. 2019, p. 212, n. 21), and for an online questionnaire, a video call with screen sharing works too. Cognitive interviews reveal what goes on in people's heads between the question and the answer (Schnell 2019, p. 131).

In the field: standard pretest and respondent debriefing

In a standard pretest, test participants fill in the questionnaire under conditions as realistic as possible, and you measure how long it takes. Because you don't ask follow-up questions, it gives you little feedback on its own (Jacob et al. 2019, p. 215). Add a respondent debriefing: after the run, the test participant tells you what they noticed about each question. That way, you get a realistic estimate of completion time and participants' comments in one session (Jacob et al. 2019, p. 215). The debriefing can also be done in writing: after the questionnaire, the test participant fills in a short second questionnaire. You'll find a ready-made pretest template for this at QUESTIONSTAR.

Malhotra and Birks recommend doing the first test in person, because you only see hesitation if you're there, and then testing in the mode planned for the main survey. For an online survey, that means testing online (Malhotra and Birks 2007, p. 391).

Cognitive interviews: how to ask follow-up questions

In a cognitive interview, you do the opposite of what's allowed in a real survey: you ask follow-up questions and encourage comments. Jacob et al. describe several techniques you can combine (Jacob et al. 2019, pp. 213-214).

Think aloud. While answering or right afterward, the test participant says everything that goes through their mind. That's demanding for both sides, takes time and is only worth it for selected, difficult questions. Many people give reasons rather than their actual train of thought (Möhring and Schlütz 2019, p. 185). Schnell therefore recommends a short introduction with practice questions, such as "How many windows did the house you grew up in have?", before the actual questions (Schnell 2019, pp. 423-424).

Probing. After the answer, you ask targeted follow-up questions. That's easier than thinking aloud, and the more specific the probe, the more you get out of it (Möhring and Schlütz 2019, p. 185). Proven wordings from Schnell are "What made you give that answer?", "What were you thinking of when you answered?" or, for single terms, "What do you understand by 'household' here?". One of his examples: for a question about medical treatment in the last six months, a probe clarifies whether respondents also count self-medication (Schnell 2019, p. 136). Schnell argues that such probes take so little effort that they should be part of every pretest (Schnell 2019, p. 137).

Paraphrasing. The test participant repeats the question in their own words. That quickly shows you whether they understood the question the way you mean it.

Confidence rating. After answering, the test participant says how sure they are. This works best for knowledge and behavior questions. You only find out the reason for any uncertainty by asking a follow-up question (Möhring and Schlütz 2019, p. 185).

Let's see what this looks like in practice. For its course evaluation, a university tests this statement:

"The course was well organized."
very dissatisfied / somewhat dissatisfied / neither satisfied nor dissatisfied / somewhat satisfied / very satisfied

Two things stand out in the interview. The answer options don't match the statement, because "was well organized" calls for agreement, not satisfaction. And when paraphrasing, one student says: "So whether the room was okay and the slides were online on time." Another thinks of how the lectures connected to each other. Same question, two different things. The revision pins down the aspects being rated and brings the statements and the scale into line:

"The course materials were available on time."
"The sessions clearly built on each other."
strongly disagree / somewhat disagree / neither agree nor disagree / somewhat agree / strongly agree

The example is made up. How to word questions more precisely is covered in Writing questions.

Three rules for the interview: decide in advance which questions you'll explore in depth and with which probes. Address your test participants as experts who are helping you, which increases their motivation (Jacob et al. 2019, p. 214). And keep your probes neutral, because a follow-up question can influence answers too (Möhring and Schlütz 2019, p. 185).

To see what such interviews look like in real studies, take a look at the GESIS Pretest Database, with the results of numerous cognitive interviews. Jacob et al. call it a "real treasure trove" (Jacob et al. 2019, p. 218).

How to check the technical setup of an online questionnaire

An online survey adds a source of error that doesn't exist on paper: the technology. What looks perfect on your computer can fail on a phone, so the technology absolutely has to be part of the pretest (Möhring and Schlütz 2019, p. 147, more under Layout). Check the programming, the skip logic, compatibility with different browsers, the display on different devices and how respondents interact with each question format (Möhring and Schlütz 2019, p. 184).

You can do most of this on your own, before the first test participant gets the link, in four steps:

1. Run through every path once. Draw the skip logic as a flowchart and go through every path once, including the unusual ones. Anyone who answers no to "Have you ordered anything online in the last twelve months?" must not see a question about delivery time. Anyone who answers yes must see it. How to plan skip logic is covered in the section Filter questions of the questionnaire design article.

2. Look at it on real devices. Open the questionnaire on at least one phone, one tablet or laptop, and in a second browser. Watch out for matrix questions that get too wide on a phone, long answer lists, and sliders that are harder to use on a phone (Möhring and Schlütz 2019, p. 181). Responsive design, that is, a layout that adapts to the device, is something you have to check explicitly in the pretest (Möhring and Schlütz 2019, p. 179). In QUESTIONSTAR, you open the preview on your phone with a QR code, and nothing is recorded (more below).

3. Test required questions and validation. Try to leave a page without answering, type letters into a number field, select every option in a multiple-choice question. Every error message should say clearly what to do.

4. Check test data against your analysis. Fill in the questionnaire a few times with made-up answers, export the data and put it next to your empty analysis tables. If an answer fits into no table, the question may be unnecessary. If a table stays empty, a question is missing (Malhotra and Birks 2007, p. 392).

But the technical test doesn't replace testing with people: you know what you mean, so you never get stuck where others do.

Pretest sample size: how many people do you need?

There's no universally required sample size, and there are no mandatory standards for how to carry out pretests either (Schnell 2019, p. 126). The number depends on what you're testing. Checking comprehension takes a few people and in-depth conversations. Measuring completion time or examining initial response distributions takes more people.

In my experience, even a small study should have at least three people, ideally five to ten. Three is my minimum for the comprehension interviews, not for the field test. For a thesis or a medium-sized customer survey, a realistic combination is five to ten cognitive interviews followed by a round with ten to 30 people under the same online conditions as the main survey.

Who tests matters more than how many

Your test participants must come from the same population as your later respondents (Jacob et al. 2019, p. 214). According to Schnell, apart from the very first developmental pretest, pretests should be based on a true random sample of the target population (Schnell 2019, p. 142). For a purely qualitative check in the field test, Jacob et al. consider a quota sample acceptable too (Jacob et al. 2019, p. 216). For a general population survey, surveying students is by no means enough (Jacob et al. 2019, p. 215). Schumann also advises against homogeneous groups such as students or a company's employees, unless they're exactly your target group (Schumann 2019, p. 76).

Colleagues, friends and family are still useful for a very first run in which everyone knows it's a test. That hardly counts as a real pretest, though, and it should be followed by a test in which people fill in the questionnaire like real respondents (Schnell 2019, p. 130).

And finally: test more than once. After every major revision, run a new test with different people, because revisions easily introduce new mistakes (Malhotra and Birks 2007, p. 392; Schumann 2019, p. 76).

What to watch for: comprehension, duration, dropouts, phones

A pretest only tells you something if you know what you're looking for. Telling observers to "note all problems" is considered inefficient (DeMaio 1983, cited in Schnell 2019, p. 138). A short list of signals that you record for every question works better.

Comprehension. Typical signs include requests for clarification, long pauses, questions that need to be repeated, and answers that don't fit the question (Jacob et al. 2019, p. 217). Paraphrasing shows you whether someone understands something different from what you mean.

Duration. Measure the time from the start to the last page for every test participant who fills in the questionnaire without interruption. If you ask follow-up questions along the way, the timing is no longer realistic (Jacob et al. 2019, p. 215). A pretest with follow-up questions may take longer, but no more than twice as long as the planned survey (Schnell 2019, p. 142). How long a questionnaire can be is covered under Length.

Dropouts and skipped questions. If someone drops out or skips a question, ask why. In computer-assisted surveys, you can see dropout rates and identify skipped questions directly in the data. Annoyance and boredom are signals too (Möhring and Schlütz 2019, p. 184).

Answer distributions. With a relatively large number of cases, it's worth looking at the distributions: categories nobody chooses, very lopsided distributions and many evasive answers such as "don't know" can point to unclear, loaded or sensitive questions, but they can also reflect the actual situation in the target group (Möhring and Schlütz 2019, p. 184). By the way, "can't judge" only belongs with questions where someone genuinely may have no knowledge or experience. The pretest shows whether you're offering this answer in the right places.

Device. Ask every test participant which device they used, and ask some of them specifically to use their phone.

How to document results and decide on changes

Many pretests end with a single sentence in the report: a pretest was carried out, and no problems occurred. Schnell scoffs that such pretests only serve to reassure the client (Schnell 2019, p. 127). So record the results in a short log. Jacob et al. even call such a pretest report a mandatory part of a scientific publication, including the size and composition of the pretest sample (Jacob et al. 2019, p. 218).

A simple table is enough. One row per problem, not per test participant:

The numbers are made up. The duration comes from runs without follow-up questions. The other rows come from an assessment sheet for each test interview that records requests for clarification, repetitions, hesitation, refusals, uncomfortable or time-consuming questions and problems with skip logic (Jacob et al. 2019, p. 217).

When to change something

Not every comment calls for a change. If problems pile up for one question, that's a sign it needs revising, and the type of problem shows you how (Schnell 2019, p. 138). Still, a single genuine misunderstanding of a question that's central to your analysis is a reason to think again. Matters of taste such as "I would have worded that differently" are worth noting, but you don't necessarily have to act on them.

If you change a lot after the test, test the new version again (Schumann 2019, p. 76). And don't add any new questions after the last test (Schnell 2019, p. 128). A question that's only added after the test is an untested question.

Pretesting in a thesis and in a company

In a thesis, a pretest is essential, at least with fellow students, ideally with people from the target group (Möhring and Schlütz 2019, p. 183). Elaborate lab methods with trained interviewers are used more for developing scales and for sensitive topics, less for student projects (Möhring and Schlütz 2019, p. 185). A few well-run conversations with follow-up questions and one online round are usually enough here. In your methods section, describe who you tested with, how many people, and what you changed afterward (Jacob et al. 2019, p. 218). How the pretest fits into the overall thesis process is covered in How to write my thesis.

In a company, what's usually missing isn't knowledge but time: the send date is fixed, the questionnaire comes back late from internal sign-off, and the pretest gets cut. Yet a small pretest is quick to do. In my experience, five customers or employees from different backgrounds are often enough, with a half-hour video call for each person. That's not much compared with a survey whose key question was misunderstood. If an agency runs the survey for you, Schnell's advice applies: run the pretests to improve the questionnaire yourself, or specify exactly how they're to be done. He considers a general assessment of whether the questionnaire causes problems completely unsuitable as a pretest (Schnell 2019, p. 414). Test the invitation too: Schnell considers repeated cognitive pretests of cover letters with people from the target group indispensable (Schnell 2019, p. 433). How invitations and reminders affect participation is covered in Survey response rate.

How to pretest in QUESTIONSTAR

A brief plug for our own tool, with a focus on specific features. No software can run a pretest for you. You conduct the interviews yourself. QUESTIONSTAR helps with the technical checks, with reviews by other people and with the field test.

Preview. The preview doesn't record any responses. The desktop, tablet and phone views are based on typical screen widths. For a real device, scan the QR code in the preview with your phone: the preview opens there, and again nothing is saved. You can choose "Show hidden questions", "Ignore skip logic" and "Ignore validation" to look through all the questions at once. To check the skip logic, switch all three off again and run through every path once.

Sharing the preview and exporting the questionnaire. You can share the link to the full-screen preview: reviewers and test participants open the questionnaire without logging in, and nothing is saved. That works for an expert review and for a cognitive interview on a video call, and you note answers and observations in your log. For comments on paper, export the questionnaire as a Word or PDF file, which also gives you the appendix for your methods section. For the questionnaire conference, you can also edit the survey together with your team.

Field test with the share link. For the test under real conditions, publish the survey and give your test participants the normal link. The completions widget shows how many people started and how many completed. The difference is dropouts or questionnaires still in progress. During the test, configure the Access settings, such as "Prevent multiple completion", as you plan to use them in the main survey, so that you test them too.

Deleting test responses. You can delete responses from the field test before the real start. That way, test data doesn't get mixed up with the data from your survey.

Template. Under Templates, you'll find a pretest template: a short questionnaire your test participants fill in after the actual questionnaire, that is, a written respondent debriefing. What QUESTIONSTAR doesn't do: it doesn't run interviews, it doesn't measure the time per question, and it doesn't decide which of your questions are misleading.

Compact pretest plan: checklist

When time or money is short, Jacob et al. name an expert pretest and a standard pretest with an assessment sheet as the minimum (Jacob et al. 2019, p. 217). The following plan adds the technical check and cognitive interviews to this minimum for online surveys:

1. Set up your analysis tables before the test begins.
2. Have one or two people with survey experience review the draft, using a checklist.
3. Run through every skip logic path yourself in the preview, on a real phone, on a tablet or laptop, and in a second browser.
4. Conduct individual cognitive interviews with five people from the target group: have each person fill in the questionnaire, probe on difficult questions and ask them to paraphrase selected questions.
5. Keep a problem log and revise the questions.
6. Have ten or more people from the target group fill in the revised version online, some of them on a phone, and record the duration and dropouts. Then have them fill in the pretest template as a written debriefing.
7. Export the test data and check it against your analysis tables.
8. Delete the test responses, add the measured duration to the introduction, and don't add any new questions.
9. Record in your report or methods section: who tested, how many people, which method, and what was changed.

Frequently asked questions about questionnaire pretests

What is a questionnaire pretest?

A questionnaire pretest checks whether a questionnaire works technically, is well designed and has suitable content before the actual data collection begins (Möhring and Schlütz 2019, p. 182). A few people from the target group fill in the questionnaire, and you check comprehension, answer options, duration, skip logic and layout.

How many people do you need for a pretest?

For cognitive interviews, five to ten people are often enough (Jacob et al. 2019, p. 214). For a standard pretest, Möhring and Schlütz give ten to 50, depending on the size of the main sample (Möhring and Schlütz 2019, p. 184). What matters more than the number is that the test participants belong to the target group.

What is the difference between a pretest and a pilot study?

The pretest tests the questionnaire. The pilot study rehearses the whole process of the main study on a small scale, including the sample, fieldwork and analysis (Schnell 2019, p. 143; Möhring and Schlütz 2019, p. 181). It's mainly worth it for large studies with new procedures.

What is a cognitive pretest?

A cognitive pretest uses cognitive interviewing to examine how respondents understand a question and arrive at their answer. Typical techniques include think aloud, targeted probing, paraphrasing and confidence rating (Jacob et al. 2019, p. 213).

Can I include the pretest responses in my analysis?

Usually not. The questionnaire has usually changed after the pretest. In addition, test participants are rarely selected the same way as the actual sample (Schumann 2019, p. 76). So delete the test responses before the start. An exception is defensible if the questionnaire didn't change after the test and the test participants were recruited exactly like the sample. In that case, say so openly in your methods section.

Do I need a pretest if I use proven questions?

Yes. Schumann considers a pretest unnecessary for questions that have already been used in representative surveys (Schumann 2019, p. 76), but a new context or a different target group can make even proven questions unusable (Schnell 2019, pp. 123-124). I side with Schnell: the test can just be shorter.

What does pretest mean in an experiment?

In an experiment, pretest means something else: the measurement before the treatment, for example before an advertising campaign, which is then compared with the measurement afterward (Malhotra and Birks 2007, pp. 313-314). It has nothing to do with testing a questionnaire.

Sources

Belson, W. (1981). The Design and Understanding of Survey Questions. Aldershot: Gower. Cited in Schnell (2019), p. 137.

DeMaio, T. (1983). Approaches to Developing Questionnaires. Technical Report 10. Washington, DC: Office of Management and Budget. Cited in Schnell (2019), p. 138.

Faulbaum, F., Prüfer, P. and Rexroth, M. (2009). Was ist eine gute Frage? Die systematische Evaluation der Fragenqualität [What is a good question? The systematic evaluation of question quality]. Wiesbaden: VS. In German. Cited in Möhring and Schlütz (2019), pp. 183-184.

Hunt, S. D., Sparkman, R. D. and Wilcox, J. B. (1982). The Pretest in Survey Research: Issues and Preliminary Findings. Journal of Marketing Research, 19(2), 269-273. Cited in Schnell (2019), p. 126.

Jacob, R., Heinz, A. and Décieux, J. P. (2019). Umfrage. Einführung in die Methoden der Umfrageforschung [Surveys: an introduction to the methods of survey research] (4th ed.). Berlin/Boston: De Gruyter Oldenbourg. In German.

Kreiselmaier, J., Prüfer, P. and Rexroth, M. (1989). Der Interviewer im Pretest. Evaluation der Interviewerleistung und Entwurf eines neuen Pretestkonzepts [The interviewer in the pretest: evaluating interviewer performance and outlining a new pretest concept]. ZUMA-Arbeitsbericht Nr. 89/14. Mannheim: ZUMA. In German. Cited in Schnell (2019), p. 127.

Malhotra, N. K. and Birks, D. F. (2007). Marketing Research. An Applied Approach (3rd European ed.). Harlow: Pearson Education.

Marx, P. (2026). Principles of Survey Research. Online course, slide 115 (chapter 3). QUESTIONSTAR. Course, chapter 3.

Möhring, W. and Schlütz, D. (2019). Die Befragung in der Medien- und Kommunikationswissenschaft. Eine praxisorientierte Einführung [Surveys in media and communication studies: a practical introduction] (3rd ed.). Wiesbaden: Springer VS. In German. DOI 10.1007/978-3-658-25865-8.

Schnell, R. (2019). Survey-Interviews. Methoden standardisierter Befragungen [Survey interviews: methods of standardized surveys] (2nd ed.). Wiesbaden: Springer VS. In German. DOI 10.1007/978-3-531-19901-6.

Schumann, S. (2019). Repräsentative Umfrage. Praxisorientierte Einführung in empirische Methoden und statistische Analyseverfahren [Representative surveys: a practical introduction to empirical methods and statistical analysis] (7th ed.). Berlin/Boston: De Gruyter Oldenbourg. In German.

Willis, G. B. (2005). Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks: Sage. Cited in Schnell (2019), pp. 125 and 132-133.

I hope you find test participants who tell you honestly where they stumbled, and that you have the courage to cut a favorite question if nobody understands it.

Related articles:

• Questionnaire design: structure, question types and checklist
• Writing questions: how an idea becomes a robust question
• How to write my thesis

Dr. Paul Marx

Author: Dr. Paul Marx
Dr. Paul Marx is the founder and Managing Director of QUESTIONSTAR and was a professor of marketing at the University of Siegen.