Open-ended questions: how to ask and analyze them

When open-ended questions are worth asking, how to word them, and how to code, count and report open-ended responses. With a worked example and AI tips.

An open-ended question offers no answer options, so respondents answer in their own words (Möhring and Schlütz 2019, p. 73). It's worth asking when you don't yet know the possible answers, when answer options would influence the result, or when there are a great many possible answers (Schnell 2019, p. 69). To analyze open-ended text responses, you usually sort them into categories afterward (Schnell 2019, p. 72) and count them, ideally with a codebook and a second person as a check.

Let's go through it step by step. You'll learn:

• when open-ended questions are useful and when closed-ended questions are the better fit,
• how to word open-ended questions so that you get usable answers,
• how to code open-ended responses into categories, with a worked example,
• how to check whether the coding is reliable, and how to report and quote the results,
• what AI can do when coding and where you have to check for yourself.

Clients tend to love open-ended questions, and analysts tend to dread them. With a few rules, free text turns into results you can rely on.

Open-ended vs. closed-ended questions: when are open-ended questions useful?

Möhring and Schlütz point out that the answer is open, rather than the question. No answer options are given, and respondents can say whatever they like within a given frame (2019, p. 73). A closed-ended question, by contrast, presents the answers to choose from along with the question text (ibid., p. 75).

Schnell names three reasons for an open-ended question: the possible answers really are unknown in an exploratory study, the answer options must not influence response behavior in any way, or a very large number of possible answers is expected (2019, p. 69). Möhring and Schlütz add that open-ended questions are a stricter test of knowledge. Asked what Esperanto is, 35% gave the right answer to an open-ended question, compared with 52% when answer options were offered (Noelle-Neumann and Petersen 2005, cited in Möhring and Schlütz 2019, p. 73). Open-ended questions also capture spontaneous associations and whatever is "top of mind", and they show respondents that their personal opinion counts (ibid., p. 74).

There are serious drawbacks on the other side. Open-ended questions demand verbal skill and motivation, so the answers are often meager, and the analysis takes a lot of effort (Möhring and Schlütz 2019, p. 75). According to Malhotra and Birks, open-ended questions implicitly give more weight to articulate respondents (2007, p. 381). In web surveys, open-ended questions also raise the risk of breakoff (Schnell 2019, p. 71).

My course sums up the trade-off like this: unlimited possibilities versus complex coding (Marx 2026, slide 81).

A partially open (or hybrid) question combines the two formats. It offers answer options and adds "Other, please specify" with a text field (Jacob et al. 2019, p. 105). Because usually only a few respondents choose this option, the coding effort stays manageable (Schumann 2019, pp. 60-61).

In practice, this means that in a survey with many respondents, the closed-ended question is the default (see Questionnaire design). Use open-ended questions selectively: to explore, to ask for reasons after a critical rating, to catch what's missing from a list, and as a closing question (Malhotra and Birks 2007, p. 381).

Explore with open-ended questions first, then measure with closed-ended ones. An open-ended question shows which topics exist. How widespread a topic is can be measured better with a closed-ended question. That's why an open-ended question is often combined with a closed-ended one that follows it (Möhring and Schlütz 2019, p. 74). Or you ask the open-ended question in the pretest and build the list for the main survey from the answers.

How do you word open-ended questions?

In a closed-ended question, the answer options explain what's meant. An open-ended question lacks this help, so the question text itself has to be precise (Möhring and Schlütz 2019, p. 74). Jacob et al. speak of the "multi-dimensionality" of open-ended questions: it's largely left to respondents what they refer to (2019, p. 143). Their example: in response to "Where do you live?", someone may enter the federal state, the district, the municipality or the exact address. If you want to know the place of residence, ask "What is the name of the town or city you live in?" (ibid.).

Six rules help.

One subject per question. Ask about praise and criticism separately. Otherwise, the answers are hard to separate later.

Specific rather than general. "Do you have any comments?" invites everything and nothing. As a closing question, it's a useful outlet (Malhotra and Birks 2007, p. 381). But if you want to know something specific, ask about it. Give a topic and a direction, but not an answer.

Examples when the kind of answer is unclear. If you ask about someone's favorite wine, do you mean the color, the origin or the grape variety? Examples clarify the frame of reference and serve as a memory aid (Tourangeau et al. 2017, cited in Möhring and Schlütz 2019, p. 74). Because respondents take their cue from examples, the answers are easier to summarize (Möhring and Schlütz 2019, p. 74). So choose examples that show the kind of answer you want, not a particular topic.

Careful with "why?". Jacob et al. advise against why questions because they put respondents under pressure to justify themselves (2019, p. 154). Möhring and Schlütz, on the other hand, see asking for reasons as one of the purposes of open-ended questions (2019, p. 74). A workable middle course: ask for reasons when you really need them, not as an interrogation but tied to a specific occasion (third row of the table below).

Use sparingly and keep optional. Möhring and Schlütz advise keeping open-ended questions rare, because the effort often outweighs the value of the answers (2019, p. 189). Limit yourself to the open-ended text questions you'll actually analyze, and don't make any of them required. Otherwise, people with nothing to say will write "nothing" or "xxx", or break off.

The right place. A follow-up question about reasons goes directly after the rating it refers to. A general open-ended question goes at the start of a topic or at the end of the questionnaire.

You'll find general rules for good questions in the article Writing questions.

How to analyze open-ended responses: coding answers into categories

Analyzing open-ended text responses almost always means sorting them into categories afterward (Schnell 2019, p. 72). Open-ended numeric answers, by contrast, can be analyzed directly as numbers (Jacob et al. 2019, p. 183). Schumann describes the principle: each open-ended answer is assigned a number that stands for an answer category. After that, the question can formally be analyzed like a closed-ended one (2019, p. 59). Different wordings end up in the same category. Schumann's examples (translated here), "We have to protect the environment from destruction!" and "Environmental pollution has to be stopped!", both go under "environmental protection" (ibid., n. 98). The categories formed this way are nominal: you can count frequencies, but you can't calculate a mean from the code numbers (Weinreich and von Lindern 2008, p. 107, see Levels of measurement).

If no suitable category scheme exists, one has to be developed (Schumann 2019, p. 59). So there are two routes:

• Defined in advance: The categories come from the research question, an earlier wave or a model. Advantage: comparable across waves, as long as definitions and assignment rules stay the same. New categories may mean recoding earlier answers. Risk: you overlook what you didn't expect.
• Developed from the answers: A group of subject-matter experts reads at least part of the answers, at least 50 (or all of them if there are fewer), discusses the content and forms categories from it (Weinreich and von Lindern 2008, p. 106). Advantage: you find new things. Risk: every wave ends up with a different scheme.

In methods textbooks, you'll find the two routes under the names deductive and inductive category development. In practice, people mix the two: you start with the categories you expect and add what the answers show.

Malhotra and Birks give two rules for this. First, the categories should be mutually exclusive and collectively exhaustive. An "other" category makes the scheme exhaustive, but no more than 10% of the answers should fall into it. Second, important issues get a category of their own even if nobody mentions them, because that's a finding too (2007, pp. 484-485). To give a sense of scale: 1,000 differently worded answers may come down to about 20 issues (ibid., p. 484).

One code or several? Malhotra and Birks and Schumann assume one code per answer. In practice, an answer often contains more than one topic: "Shorter wait times and better-trained staff" raises two. The "mutually exclusive" rule then applies to each statement: every single topic belongs in exactly one category, while the answer as a whole can receive several codes. Weinreich and von Lindern explicitly provide for an answer being assigned to several categories (2008, p. 107). Technically, you then create a separate column for each category, as you would for a question that allows multiple selections, with 1 for "applies" and 0 for "doesn't apply" (Möhring and Schlütz 2019, p. 189; Jacob et al. 2019, p. 182, n. 17). If you also need exactly one value per answer, define a primary code, for example the topic mentioned first. In the example below, the frequencies count all codes, and the agreement check uses the primary code.

Schumann also recommends making the scheme as fine-grained as possible: fine categories can later be combined into broader ones, depending on the research question (2019, p. 59, n. 99).

Everything you decide along the way belongs in a codebook, the collection of coding instructions (Malhotra and Birks 2007, p. 483). For open-ended questions, this format per category is enough:

Add two more codes: "other" for content without a category of its own, and "not usable" for answers without content, such as "-" or "xxx". Keep the two apart: "other" contains content you can analyze, "not usable" doesn't.

Do you always have to code? With 20 to 50 short answers, a sorted list is often enough, according to Weinreich and von Lindern. With more answers, you lose track in a list, and if you want to relate the topics to other questions, you have to code (2008, p. 106).

Worked example: 24 answers, six categories

An invented example makes the steps concrete. A company surveys 40 customers and ends with the optional question "What should we improve first about our customer service?". 26 of them write something, which is 65%. Two answers aren't usable ("-" and "don't know"), leaving 24 answers to code. The codebook has six categories, each with a code letter: E Ease of contact and response time, K Knowledge and resolution, R Request updates and information, F Friendliness and tone, L Leave as is (praise or nothing to improve), S Something else (other).

Some of the answers and their codes:

Here, E, K, R and F capture a need for improvement, even when it's phrased as a wish. L captures only purely positive answers or the statement that nothing needs improving. That's why the third answer gets only K: it contains "friendly", but it criticizes the missing solution. Two of the 24 answers get two codes, so there are 26 mentions in total. The result:

Three things stand out. The "% of answers" column adds up to 108.3% because two answers contain two topics. At 8.3%, "other" stays below the 10% ceiling (Malhotra and Birks 2007, p. 485). And the same eight mentions of ease of contact come to 20.0%, 30.8% or 33.3%, depending on the base. Which number is the right one is covered in the section on reporting.

How do you check whether the coding is reliable?

With an open-ended question, a human decides which category an answer belongs to, and two humans don't always decide the same way. Schumann points out that objectivity of analysis can be lacking precisely when open-ended answers are coded (2019, p. 29, n. 20): not everyone who analyzes the answers arrives at the same result (see Reliability and validity).

The practical answer is a second person. Weinreich and von Lindern recommend having at least two people code the answers independently, comparing the assigned codes and recoding the discrepancies, often with a third person who decides (2008, p. 107).

In the example, coders A and B independently assign one primary code to each of the 24 answers. For 21 answers, both choose the same code, which is 87.5% agreement. For three answers, they differ:

The 87.5% doesn't yet take into account how much agreement you'd expect purely from how often each category is used. Cohen's kappa corrects the observed agreement for this chance expectation (Cohen 1960). The formula:

A kappa of 1 means perfect agreement, and 0 means no more than chance. Weinreich and von Lindern speak of good agreement at values above 0.7 (2008, p. 107). At 0.84, the example exceeds that threshold. But with only 24 answers, the value should be read with caution. A simplified calculation that treats the chance expectation as fixed gives a standard error of about 0.08. It illustrates the uncertainty, but it doesn't provide a dependable confidence interval for kappa. So take the 0.84 as an indication, not as proof.

This kappa refers to the primary code, that is, to exactly one code per answer. If you code several topics per answer, calculate kappa separately for each category from its 0/1 column.

More important than the number is what you do with the discrepancies. Each one points to a place where the codebook is unclear. In the example, the three rules from the right-hand column go into the codebook. After that, both coders check all answers that the changes might affect and recode them where necessary.

With 200 comments, you don't have to double-code everything. A second person first codes a sample, about 50 answers as a rule of thumb. You resolve the discrepancies, sharpen the codebook and then code the rest. In theses and study reports, the agreement, including the statistic, belongs in the methods section.

One warning from Malhotra and Birks belongs here: if you only code what is unambiguously on the surface, you may achieve reliability at the expense of validity (2007, p. 251).

How do you report open-ended responses?

Three things matter in the report: the base of the percentages, the limits of a frequency, and the right place for quotes. With the 0/1 column per category, you can also cross-tabulate the topics with other questions (see Weinreich and von Lindern 2008, p. 106).

State the base. A good default is "% of people with a usable answer", here 33.3%, because this number shows how many of those who said something mention a topic. State the base along with it: "8 of the 24 people who gave a usable answer". With small numbers, absolute values are more honest than percentages with decimals: "8 of 24" doesn't promise a precision that isn't there.

Frequency isn't prevalence. That 8 of 40 people mention ease of contact doesn't mean that only 20% think it needs improving. People who write something can differ systematically from those who write nothing: in a survey of 661 people in a military organization, it was mainly the dissatisfied who commented (Poncheri et al. 2008, cited in Schnell 2019, p. 71). Linke reports something similar from employee surveys (2018, p. 40). And not everyone who has a problem mentions it spontaneously. How widespread a topic is can be measured with a closed-ended question (see also Employee survey).

Quotes as illustration, not as proof. Weinreich and von Lindern describe how free text draws attention as if by magic, even when a chart represents 300 respondents and a quote represents three or four. Coding puts free-text responses in perspective, often giving them a supporting role in the analysis (2008, p. 108). Quotes illustrate a category whose size you've already shown in numbers. Choose typical ones, not the most drastic.

A results line in the report can look like this:

Quoting answers responsibly

Open-ended answers are the part of a survey where anonymity is lost most easily. A single scale value rarely reveals who answered. A sentence like "Since I started working as the only woman in the warehouse ..." can be unmistakable in a small company.

Wiertalla describes an employee survey in which, to protect the writers, comments were analyzed only for managers who led at least 50 direct and indirect employees, which generally meant only at executive board and director level. The employees weren't aware of this rule (2013, p. 157). Malhotra and Birks call for special care when sample details are too revealing and when verbatim quotations appear in reports, particularly in employee research (2007, p. 425). Five rules follow from this:

1. No quotes from small groups. Don't show quotes by department, team or location when the group is small. Set the threshold in advance and stick to it. For verbatim quotes, it should be higher rather than lower than for scale values, because text is easier to recognize.
2. Remove identifying details. Delete names, places, dates, rare job functions and very personal circumstances, and mark the spot with [...] or a neutral placeholder such as [name].
3. Don't change the meaning. Shortening is fine, rewording isn't. You may correct typos if someone would otherwise be recognizable by their writing style.
4. Don't combine. An inconspicuous quote becomes conspicuous when "Purchasing department, over 55, manager" appears next to it.
5. Announce what will happen. If you want to pass on answers verbatim, say so in the introduction, and say who will read the comments. In Wiertalla's case, that's exactly what many employees never learned (2013, p. 157).

If respondents write names, health information or contact details into a text field, that's personal data, even in an otherwise anonymous survey (see Data protection in online surveys). How to protect small groups in reports is covered in the article on social desirability bias.

Coding open-ended responses with AI: what works and what doesn't

The methods literature this article draws on is older than today's language models. What follows is therefore my assessment and way of working, not a finding from the literature.

Weinreich and von Lindern wrote about the programs of their time that they couldn't understand text and had to be trained by human coders (2008, pp. 107-108). Linke writes that in practice you regularly find that automated coding has even changed the meaning of a comment (2018, p. 40). How well today's models code your data is something you'll only know once you've checked it on a sample. Six rules:

How much control you need depends on the purpose. In my assessment, today's language models do a useful job of coding free-text responses. In practice, for example with customer feedback or an internal survey, you can therefore leave the coding largely to the AI if the sample from rule 3 shows good agreement. For academic work, a stricter standard applies: there, a human codes alongside the AI, and you disclose how you proceeded.

There are three limitations to keep in mind.

For academic work, the second person remains the benchmark. Even when you and the AI agree well, independent coding by a second human remains the benchmark from the section on agreement. State who or what did the coding.

The result isn't automatically reproducible. A language model may code the same answer differently in a second run, and models are replaced all the time. Your result is therefore the saved table of codes, together with the model, date, instruction and codebook (rule 6). If you want to know for sure, have a sample coded twice and compare.

Free text can contain personal data. Before you hand answers to an AI service, check which answers it will get to see and whether you're allowed to pass the data on to it (for QUESTIONSTAR, see the next section).

Open-ended questions in QUESTIONSTAR

QUESTIONSTAR doesn't do the thinking for you when it comes to coding, but it makes it easier to ask open-ended questions and prepare the responses for analysis.

In the export, the sum of a 0/1 column is the number of mentions. Divided by the number of usable answers, it gives the share. In SPSS, you cross-tabulate the columns with the other questions. There's more on the word cloud in the article Analyzing online surveys.

With an AI assistant. If QUESTIONSTAR is connected to an AI assistant via the MCP interface, for example in ChatGPT or Claude, the assistant reads your survey's answers directly and can code them by your codebook. The assistant does the coding, not QUESTIONSTAR. According to the help page, the MCP server doesn't return respondents' names, email addresses and similar contact data, and the AI provider is a separate processor whose data processing agreement you should review separately (ibid.). In addition, the following applies in QUESTIONSTAR: questions you've marked as personal data in the survey aren't passed to the assistant when that setting is switched on, even if they're text questions. Check free-text responses yourself if the question isn't marked this way.

Private questions. You mark questions that collect personal data as "Private question" in the question settings (see privacy settings). Their answers are shown in the analysis and in the raw data export only once at least ten questionnaires have been fully completed (ibid.). A text question in which you expect names or very personal information should be marked this way. Decide this before you launch: once the survey has been started for the first time, the marking can't be removed, and you can only mark additional questions (ibid.). The threshold of ten questionnaires applies to the survey as a whole. It doesn't protect small departments in a filtered report.

Checklist: asking and analyzing open-ended questions

1. For each open-ended question, clarify the reason: exploration, reasons, a great many possible answers, or a frequency? If none of these fits, ask a closed-ended question.
2. One subject per question, a specific topic, and if necessary an example of the kind of answer. Few open-ended text questions, optional, in the right place.
3. When most possible answers are known but some may be missing, use a partially open question with "Other, please specify".
4. Set up a codebook: name, definition, boundary, anchor examples, rule for multiple topics, "other", "not usable".
5. Keep categories fine-grained and clearly separated. Each individual statement should fit exactly one category. Together, the categories should cover everything, with no more than 10% of answers falling into "other".
6. Have a second person code a sample independently, calculate agreement and kappa, and sharpen the codebook. Check AI coding the same way, and record the model, date, instruction and codebook.
7. Count with a base: "x of n people with a usable answer", plus how many saw the question and how many wrote nothing. Don't read frequency as prevalence.
8. Choose quotes that are typical rather than drastic, with the number alongside, without identifying details, and not from small groups.

Frequently asked questions about open-ended questions

What is an open-ended question?

An open-ended question offers no answer options. Respondents answer in their own words, as text or as a number. So it's the answer that is open, not the question (Möhring and Schlütz 2019, p. 73).

What are examples of open-ended questions?

As text: "What should we improve first about our customer service?" As a number: "On how many of the past seven days did you use the app?" Partially open: a list of contact channels with the option "Other, please specify" and a text field.

What are the advantages and disadvantages of open-ended questions, and when are they worth asking?

Open-ended questions influence the answers less and give you respondents' own words. But they favor articulate respondents (Malhotra and Birks 2007, p. 381), take a lot of effort to analyze, and raise the risk of breakoff online (Schnell 2019, p. 71). They're worth asking when the possible answers are unknown, when answer options would influence the result, or when there are a great many possible answers (Schnell 2019, p. 69).

How do you analyze open-ended responses?

Open-ended text responses are sorted into categories and then counted (Schnell 2019, p. 72). To do this, you set up a codebook with clearly defined, clearly separated categories, assign each statement to one category, and report the frequencies with their base. A second person should code at least a sample independently (Weinreich and von Lindern 2008, p. 107).

What is a category scheme for open-ended responses?

A category scheme defines the groups into which open-ended answers are sorted. It's either defined in advance (deductive) or developed from the answers (inductive). The categories should be mutually exclusive and collectively exhaustive, with "other" holding no more than 10% (Malhotra and Birks 2007, pp. 484-485). If an answer contains several statements, this applies to each statement.

How do you check agreement between two coders?

Two people code the same answers independently. Cohen's kappa corrects their observed agreement for the share that would be expected by chance alone (Cohen 1960). Weinreich and von Lindern speak of good agreement at a kappa above 0.7 (2008, p. 107).

Can AI analyze open-ended responses?

AI can suggest categories and code answers by a codebook. Writing the codebook, checking a sample and verifying the quotes remain human tasks. A language model can make mistakes without anyone noticing, and it can code differently in a second run. So record the model, date, instruction and codebook, and save the table of codes. Check free text for personal information before you hand it to an AI.

Sources

Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1), 37-46. DOI 10.1177/001316446002000104.

Jacob, R., Heinz, A. and Décieux, J. P. (2019). Umfrage. Einführung in die Methoden der Umfrageforschung [Surveys: an introduction to the methods of survey research] (4th ed.). Berlin/Boston: De Gruyter Oldenbourg. In German.

Linke, R. (2018). Mitarbeiterbefragungen optimieren. Von der Befragung zum wirksamen Management-Instrument [Optimizing employee surveys: from survey to effective management tool]. Wiesbaden: Springer Gabler. In German.

Malhotra, N. K. and Birks, D. F. (2007). Marketing Research. An Applied Approach (3rd European edition). Harlow: Pearson Education.

Marx, P. (2026). Principles of Survey Research. Online course, slide 81 (chapter 3). QUESTIONSTAR. Course, chapter 3.

Möhring, W. and Schlütz, D. (2019). Die Befragung in der Medien- und Kommunikationswissenschaft. Eine praxisorientierte Einführung [Surveys in media and communication studies: a practical introduction] (3rd ed.). Wiesbaden: Springer VS. In German.

Noelle-Neumann, E. and Petersen, T. (2005). Alle, nicht jeder. Einführung in die Methoden der Demoskopie [All, not everyone: an introduction to the methods of opinion polling] (4th ed.). Berlin: Springer. In German. Cited in Möhring and Schlütz (2019), p. 73.

Poncheri, R. M., Lindberg, J. T., Thompson, L. F. and Surface, E. A. (2008). A Comment on Employee Surveys: Negativity Bias in Open-Ended Responses. Organizational Research Methods, 11(3), 614-630. DOI 10.1177/1094428106295504. Cited in Schnell (2019), p. 71.

Schnell, R. (2019). Survey-Interviews. Methoden standardisierter Befragungen [Survey interviews: methods of standardized surveys] (2nd ed.). Wiesbaden: Springer VS. In German. DOI 10.1007/978-3-531-19901-6.

Schumann, S. (2019). Repräsentative Umfrage. Praxisorientierte Einführung in empirische Methoden und statistische Analyseverfahren [Representative surveys: a practical introduction to empirical methods and statistical analysis] (7th ed.). Berlin/Boston: De Gruyter Oldenbourg. In German.

Tourangeau, R., Sun, H., Conrad, F. G. and Couper, M. P. (2017). Examples in Open-Ended Survey Questions. International Journal of Public Opinion Research, 29(4), 690-702. Cited in Möhring and Schlütz (2019), p. 74.

Weinreich, U. and von Lindern, E. (2008). Praxisbuch Kundenbefragungen [Customer surveys: a practical handbook]. München: mi-Fachverlag, FinanzBuch Verlag. In German.

Wiertalla, K. (2013). Der Einsatz von Mitarbeiterbefragungen als Lerninstrument in Unternehmen. Dargestellt anhand des Praxisbeispiels der Mitarbeiterbefragung bei der Ingram Micro Distribution GmbH [Employee surveys as a learning tool in companies, illustrated by the employee survey at Ingram Micro Distribution GmbH]. Dissertation, University of Siegen. In German.

I wish you answers that surprise you, and a codebook that puts them in order.

Related articles:

• Questionnaire design: structure, question types and checklist
• Analyzing online surveys: it starts before you collect data
• Employee survey: process, questions and analysis

Dr. Paul Marx

Author: Dr. Paul Marx
Dr. Paul Marx is the founder and Managing Director of QUESTIONSTAR and was a professor of marketing at the University of Siegen.