A/B test in a survey: does the change really work?

How to run an A/B test in a survey: compare two versions by random assignment, plan the group size, and interpret the difference using a confidence interval.

Show each of two groups one version, and let a random draw decide who gets which. Then the groups differ, on average, only in the one thing you changed. If their answers differ, and by more than chance alone would lead you to expect, that's evidence that the change caused the difference. A comparison like this is called an experiment, or a survey experiment when it runs inside a survey. With two versions it's also called an A/B test. Random assignment is its defining feature (Schnell 2019, p. 54; Malhotra and Birks 2007, p. 312).

The article proceeds in this order:

• Why isn't a comparison without random assignment enough?
• What can you test in a survey?
• How do you plan the test? One change, fixed rules before the start, group size.
• How do you read the result? Five steps from the answers to the result, with a worked example you can recalculate.
• Which mistakes make the result worthless?
• How do you set up the test in QUESTIONSTAR?
• For reference: checklist and frequently asked questions

Many people who run surveys know the situation: the team argues about two wordings of a question, marketing argues about two descriptions of a new offer, and the thesis contains the hypothesis that the order of the answer options shifts the result. There's no shortage of opinions. An A/B test replaces them with a comparison that anyone can check. What a p-value is and what a confidence interval is are explained in separate articles. This one is about how the comparison is set up, because no statistics can repair that afterward.

Why isn't a comparison without random assignment enough?

An online retailer of office supplies wants to introduce a refill subscription and has two descriptions for it. The retailer wants to know which one convinces more customers. The obvious route would be this: version A goes to the first half of the customer list on Tuesday morning, version B to the second half on Thursday evening. With B, more respondents say they would sign up for the subscription. So is B the better description?

This comparison can't tell you. The two groups differ in more than the description. On Thursday evening, different people may respond than on Tuesday morning. And if the list was sorted by customer number, the two halves contain different customers:

Only the difference in the first row is intended.

Each of the three other differences can shift the answers just as the description can. The result then combines several possible causes in a single number, and no calculation can separate them afterward. The literature calls such a result confounded and draws a clear conclusion: as long as other influences aren't controlled, nothing can be said about the effect of the change (Malhotra and Birks 2007, p. 307).

Couldn't you make the groups equal by hand, by making sure both contain the same number of regular customers and the same number of individual customers? That works only for the few characteristics you think of and know about. On all the others, the groups can still be unequal (Malhotra and Birks 2007, p. 311). Who distrusts subscriptions on principle? Who has just had a bad experience with a delivery? No customer list records that. And there's another argument against it: if you assign people by your own judgment, or let respondents choose for themselves, you easily create groups that differ even before the test (Malhotra and Birks 2007, p. 310).

You know the solution from playing cards. Shuffle a deck thoroughly and deal the cards alternately into two piles. Afterward, both piles contain about the same number of red cards, about the same number of face cards, and about the same number of high and low cards, although nobody paid attention to any of that. This even holds for properties nobody thinks of while dealing: the cards with a bent corner are spread about evenly too. The reason is simple. Shuffling knows nothing about the cards. So it can't systematically favor either pile for any kind of card. Every card has the same chance of landing on either pile, and with many cards of a kind, the chance differences largely even out. Without shuffling, it goes wrong: if you simply split a deck that is still sorted by color down the middle, all the red cards end up in one pile. It's similar with a customer list that is sorted by customer number and split down the middle.

It's similar with a list of people. If a random draw decides who gets version A and who gets version B, the draw knows nothing about the people: not who is a regular customer, not who responds in the evening, and not who distrusts subscriptions. So it can't systematically assign people with any of these characteristics to one group rather than the other. All characteristics are spread about evenly, the known ones and the unknown ones. One systematic difference remains, and it doesn't come from the draw but from you: the version.

This changes what a difference in the answers can mean. Without a draw, many explanations are possible, and you don't even know all of them. With a draw, two remain: the version, or chance in the draw. And how much chance alone accounts for can be calculated. So the draw replaces differences you don't know about with chance, which you can calculate.

"About the same" doesn't mean "exactly the same". How well the draw evens things out depends on the numbers: the smaller the list, the more likely the two groups are to turn out unequal.

The draw makes groups equal only on average and can fail with small numbers (Malhotra and Birks 2007, p. 311). That's why every test involves the question of group size, and why it's worth checking after the test whether the groups really turned out similar on the characteristics you know (ibid.). The analogy breaks down in one respect: cards stay in their pile, but people can leave their group by breaking off the questionnaire. What follows from that is covered under How do you read the result?

This random assignment is called randomization. An experiment is a comparison in which you change one thing yourself, measure its effect and control all other influences, preferably by random assignment (Malhotra and Birks 2007, pp. 306, 311). Random here means: neither you, nor the respondents, nor any characteristic of the people decides which group someone is in. With two versions, it's called an A/B test. In survey research, the same design is called a split-ballot experiment: randomly formed subgroups get questions that are the same in content but designed differently (Jacob et al. 2019, p. 46; Möhring and Schlütz 2019, p. 19).

In short: without a draw, two groups differ in many ways, and you don't know which. With a draw, they differ on average only in the version, and what chance in the draw adds can be calculated. The more people, the better the draw evens things out. That's why only an experiment with random assignment answers the question of whether a change has an effect.

The flip side holds too: if data don't come from a controlled experiment, you can only claim that a relationship between two variables exists, not in which direction it works (Marx 2026, slide 209). More on this in the article Correlation vs. causation.

What can you test in a survey?

You can test anything you can present to two groups in two versions in the questionnaire. A/B tests are also familiar from websites and newsletters. This article is about the test inside a survey.

Questions and scales. A well-known experiment from a survey of the adult population in Germany shows how much even small details can affect responses. Respondents were asked how successful they had been in life so far, on an eleven-point scale from "not at all successful" to "extremely successful". Some respondents saw the points numbered 0 to 10, the others −5 to +5. On the scale from 0 to 10, 34% chose a value from 0 to 5. On the scale from −5 to +5, only 13% chose a value from −5 to 0, that is, the same part of the scale (Schwarz et al. 1991, p. 572, also cited in Jacob et al. 2019, p. 46). The words were the same. Only the numbers next to them differed. Another example and guidance on improving question wording are in the article Writing good survey questions, and the effect of order is covered in the article Order effects in surveys.

Such tests work in the pretest as well as in the main survey (Schnell 2019, p. 135). In the main survey they come at a price: a question you ask in two versions is afterward no longer available in the same form for all respondents. To analyze responses to a question across the whole sample, you must have asked everyone that question in the same way (ibid.). If you still want to analyze that topic across the whole sample, add a question about it that everyone receives in the same form (ibid.). An A/B test doesn't replace a pretest: whether a question is understood at all is something you clarify beforehand.

Concepts and prices. Each group sees one description, one ad or one price and then answers the same questions. Such survey experiments are true experiments. But what they study is almost always reactions in words to stimuli in words, and whether these carry over to behavior outside the survey is fundamentally open to question (Schnell 2019, p. 56). Textbooks discuss this under the heading of external validity: whether a result can be generalized beyond the experimental situation (Malhotra and Birks 2007, p. 308). So the result tells you which description gets a more favorable response in the questionnaire. It doesn't tell you how many people will really sign up for the subscription. For prices, this holds even more: in a questionnaire, a "yes" costs nothing.

Each person sees only one version. Anyone who sees both compares them, and an earlier answer influences the later one (Malhotra and Birks 2007, p. 309). Then you're measuring the comparison and no longer the effect of each version by itself.

How do you plan the test?

An A/B test is planned before it starts, not during the analysis. Five steps are enough:

One change, everything else the same

The two versions differ in exactly one thing. Everything else stays the same: the same questionnaire, the same place in the questionnaire, the same questions afterward, the same fieldwork period. If you change the text and the image at once, all you know in the end is that the package works differently, but not which part.

That isn't a ban. You may test a whole package, for example the old description of an offer against a newly written one. But then the result reads "the new description convinces more respondents" and not "the shorter headline convinces more respondents". If you want to judge two things separately, such as text and image, you need all four combinations and therefore four groups. Such a design also shows whether the two changes together work differently than each one alone (Malhotra and Birks 2007, p. 321). But it needs correspondingly more people.

In practice, one of the two versions is often what you've been using so far. The control group is then the group with the current version, not a group that gets nothing at all (Malhotra and Birks 2007, p. 314).

What is fixed before the start

Write down four things before the first response comes in:

1. The outcome measure. One main question on which you compare the versions, and how you count it. In the retailer's example: the share of respondents who would be "somewhat likely" or "very likely" to sign up for the subscription.
2. The smallest difference that matters. What is the smallest difference, in percentage points, that would make you choose the new version? This number determines the group size. The table and a rule of thumb for it are under How large do the groups have to be?
3. The end. The number of responses or the date on which you analyze. Why that matters is covered under interim results.
4. Who counts. The criteria for considering a questionnaire complete and usable, applied equally to both groups.

That sounds like bureaucracy, but it takes a quarter of an hour and protects you from an obvious way of fooling yourself: searching after the test until a difference shows up somewhere.

How large do the groups have to be? Sample size for an A/B test

The smaller the difference you want to detect, the more people you need, and many more at that. The table shows this for a starting point at which 40% of respondents agree with the current version:

My own approximate calculation for the comparison of two proportions.

This is calculated as in the article Survey incentives. There, two groups test whether a voucher increases participation, and for 25% versus 35% the result is about 329 people per group. The requirement is the same: the test should show a difference of this size that really exists in about 80 out of 100 such trials, and report a difference that doesn't exist in only about 5 out of 100 trials. If you halve the difference, you need about four times as many people.

For your own numbers, a rule of thumb with the same requirements is enough:

Here p is the average of the two proportions and d is their difference, both as decimals. For 40% versus 50%, p = 0.45 and d = 0.10: 16 × 0.45 × 0.55 / 0.10² = 396, close to the 388 in the table.

Two things follow from this. First, the table counts responses, not invitations to the survey. For 776 responses, with a response rate of 20% you need about 3,900 invitations. Second, the sample size calculator answers a different question, namely how precisely a single proportion is estimated. For comparing two groups, you need more.

And what if your list has only 150 people? Then an A/B test can show only large differences: even if all 150 respond, that's 75 per group, and with a starting value of 40% that is enough only for a difference of about 23 percentage points (my own calculation, same requirements as in the table). That isn't worthless. You learn whether the new version is very much better or very much worse. Smaller differences remain open, and that's how you then read the result.

How do you read the result?

The result of an A/B test is not "B won". It's an estimated difference with uncertainty: how much the versions differ, and the range in which the true difference plausibly lies. This range is given by the confidence interval of the difference.

Five steps lead there from your responses:

If your outcome measure is a proportion, steps 2 to 4 need only two numbers per version: how many people responded, and how many of them gave the answer you're looking for. The worked example below goes through all five steps. Which test fits the comparison depends on the outcome measure: for proportions the chi-square test, for means the t-test. The article Which statistical test fits? leads you to the right one.

How the interval is calculated is shown in detail in the article P-value and statistical significance. Three cases occur:

• The interval lies entirely on one side of zero. The data clearly point in one direction. How large the lead is remains open within the interval. For the decision, the lower end matters too: would the switch still be worth it if the lead were only that large?
• The interval includes zero. The data are compatible with a lead for A, a lead for B, and no difference at all. The honest reading is "inconclusive", not "the versions are equal".
• The interval is narrow and lies entirely within the range you defined beforehand as unimportant. Then the data speak against a difference that would matter for your decision. That takes large groups.

Random assignment isn't the same as a random sample. The assignment makes the two groups comparable with each other, even if your respondents themselves weren't selected at random from all customers. The p-value and the interval then say something about the difference between the versions among the people who took part. Whether the same holds for all customers is a second question, and it depends on how the respondents were selected.

Worked example: two descriptions, 800 responses

This is how the result of the example reads in a report:

Version B (benefit first) reached 49% "somewhat likely" or "very likely", version A (terms first) 40%, with 400 randomly assigned respondents each. The difference is 9 percentage points (95% confidence interval 2.1 to 15.9 percentage points, p = 0.010). The minimum difference of 10 percentage points defined in advance lies within the interval and is therefore neither supported nor ruled out. What was measured is the stated willingness to sign up in the questionnaire, not actual sign-ups.

Three things belong with every result of an A/B test. First, the number itself: you report the difference with its confidence interval. Second, the check: did the groups turn out similar on the characteristics you know, and did more people drop out after one version than after the other? If clearly more drop out after one version, it has changed not only the answers but also who answers at all (Malhotra and Birks 2007, p. 310). Third, the scope of the conclusion: even a clean test doesn't prove the cause conclusively. Causality can never be proved, only inferred (Malhotra and Birks 2007, p. 303). But an A/B test provides the strongest basis for that inference that a survey can offer.

Which mistakes make the result worthless?

Three mistakes are hardly noticeable. With all three, the result looks clean although it isn't.

Looking at interim results and stopping early

It's tempting: after 150 responses, B is clearly ahead, the test reports p < 0.05, so you stop. The problem is that an interim result fluctuates. If you look often enough, at some point you catch a moment in which one version happens to be far ahead by chance, and if you stop right then, you take that moment for the result.

A simulation shows how much this matters. In it, both versions really work the same (45% agreement each, 400 people per group), so a significant result is always a false alarm:

My own simulation with 200,000 tests per row.

The 5 out of 100 in the first row are the risk you knowingly accept with the threshold p < 0.05. Five looks at the interim result turn that into almost three times as much.

What helps: decide before the start when you'll analyze, and then analyze once. You may check the response count at any time to see how many people have responded. Just don't look at the results for each version until the end. If you absolutely want an interim analysis, plan it in advance and apply a stricter threshold, as with many tests.

Many versions, many outcome measures, many subgroups

Every additional comparison is another opportunity for a false positive. With four versions there are already six pairs that can be compared. If all four work the same, at least one pair still shows a significant difference in about 19 out of 100 such tests (my own simulation with 200 people per version). With a single comparison it would be 5 out of 100.

The same happens if you compare two versions on ten questions, or split the result afterward by age, gender and customer group until something shows up somewhere. After such a search, "version B works with new customers under 30" is a hunch for the next test, not a result.

What helps: one main question, a few comparisons fixed beforehand, and everything else labeled as exploratory. The calculation and the correction for many tests are in the article P-value and statistical significance.

Different fieldwork periods

Two weeks of version A, then two weeks of version B: that isn't an A/B test, even if it looks like one. Between the two periods, more changes than the version. A public holiday, a price increase, a news story about your industry, the weather: events that come in from outside during a trial can shift the result just as the change itself can (Malhotra and Birks 2007, p. 308). So both versions run at the same time, and both are handed out alternately over the whole fieldwork period. Then every outside event hits both groups.

More mistakes in brief

A/B tests with QUESTIONSTAR

A word about our own product, but with concrete features.

A/B test in the questionnaire. You create all versions in the same questionnaire, as individual questions or as whole pages, and assign them to the test versions in the "AB Tests" menu. Each person then sees only one version. More than two versions are possible, and so are up to four separate tests in one questionnaire. With several tests, QUESTIONSTAR forms all combinations of the versions. The setup is described on the help page A/B Tests, where you'll also find a demo questionnaire that you can copy into your account.

Who gets which version? QUESTIONSTAR distributes the versions as evenly as possible: each new person gets the version that has been filled least so far. This rule knows nothing about the person. It depends only on when someone opens the survey. So the rule looks at the counter and not at the person. The idea from the beginning therefore holds here too: something that knows nothing about the person can't systematically assign people with a particular characteristic to one group. This shouldn't be confused with the retailer from the beginning, who sent version A on Tuesday and version B on Thursday: there, a whole period got a single version. Here, the versions alternate from person to person throughout the fieldwork period, so early and late participants alike get both versions. Whether the groups really turned out similar on the characteristics you know is something you check in the analysis, as with any test. Once the survey is running and responses have come in, QUESTIONSTAR shows a fill state, and you can share a detailed report on it with others via a link.

Analysis. In the exported data, the version is in a column of its own, with its letter (A, B, C, D ...). You export the responses as an XLSX, CSV or SPSS file and compare the groups in a statistics program. Or you have the calculation done by an AI assistant connected to QUESTIONSTAR through the MCP interface: it reads the responses and calculates the difference, its confidence interval and the test from them. The calculation is then done by the assistant, not by QUESTIONSTAR. Name the method explicitly and ask for the group sizes, difference, interval and p-value, for example like this: "Compare versions A and B on the question about the likelihood of signing up. For each version, count how many answered 'somewhat likely' or 'very likely', and report the group sizes, the difference in percentage points, the 95% confidence interval of the difference and the p-value of the test for two proportions."

Checking breakoffs per version. Because the rule fills the versions evenly, the size of the groups doesn't show you whether more people drop out after one version than after the other. The export shows that. It also contains questionnaires that weren't completed, each with the letter of its version. For each version, count two numbers: all questionnaires with that letter, and the completed ones among them. The difference is the number of questionnaires not completed per version. In the example above, the count looks like this:

The count alone doesn't yet show at which point someone dropped out. You can identify that point by checking the last question answered in the questionnaire.

A second route: random selection of pages. The help page Present a random selection of questions or pages explicitly names as a use "experiments where each person should see only one condition". For this, you create one page per version and have a set number of these pages displayed at random, for example one of three. The example on the help page is three pages with product variants, of which each person sees only one.

Not to be confused. The random order of answer options, questions or pages isn't an A/B test, even though it's also called randomization in technical language. It doesn't compare anything. It makes sure that no position in the order is favored.

Checklist: A/B test in a survey

Frequently asked questions about A/B tests in surveys

What is an A/B test (split-ballot experiment) in a survey?

An A/B test is an experiment with two versions: two randomly formed groups each get one version, for example two wordings of a question, two response scales or two descriptions of an offer. Then the answers are compared. In survey research, the same design is called a split-ballot experiment (Jacob et al. 2019, p. 46) or, more generally, a survey experiment.

Why does assignment have to be random?

Because only a random draw makes sure that the groups differ, on average, in nothing but the version, not even in characteristics you don't know about. Random assignment is the defining criterion of an experiment (Schnell 2019, p. 54; Malhotra and Birks 2007, p. 312). Without it, the groups can differ beforehand, and the difference in the answers has several possible causes.

How many participants does an A/B test need?

That depends on the difference you want to detect. If agreement is around 40%, you need about 390 people per group for a difference of 10 percentage points, a little over 1,500 per group for 5 percentage points, and just under 100 per group for 20 percentage points (my own calculation: the test should show a difference that exists in about 80 out of 100 trials, and report a difference that doesn't exist in only about 5 out of 100 trials).

Can I stop the test as soon as the result is significant?

No. If you look at the interim result several times and stop at the first significant result, you clearly increase the risk of a false alarm: with five analyses in total, including the last one, from 5 to about 14 out of 100 tests (my own simulation). Fix the time of the analysis beforehand and analyze once.

Does an A/B test need a control group?

It needs a comparison group, and it always has one: the group with the other version. If one of the two versions is what you've been using so far, that group is the control group. A group that is shown nothing at all isn't needed (Malhotra and Birks 2007, p. 314).

Can I test more than two versions?

Yes, that's then an A/B/C test. Every additional version needs a sufficiently large group of its own, and the number of false positives rises with the number of comparisons: with four versions there are six pairwise comparisons. Decide beforehand which comparisons count.

What do I do if the test shows no significant difference?

A nonsignificant difference doesn't show that both versions work the same. It only means that the data can't separate a difference from chance. Report the difference with its 95% confidence interval. A wide interval leaves the question open. A narrow interval around zero, by contrast, can speak against differences that would matter for your decision.

Does an A/B test prove that the change is the cause?

It provides the strongest basis for a causal inference that a survey can offer, but not conclusive proof: causality can only be inferred (Malhotra and Birks 2007, p. 303). And the result applies first of all to the situation in which it was measured. Whether an answer in a questionnaire matches behavior outside it is a separate question (Schnell 2019, p. 56).

Sources

Jacob, R., Heinz, A. and Décieux, J. P. (2019). Umfrage. Einführung in die Methoden der Umfrageforschung [Surveys: an introduction to the methods of survey research] (4th ed.). Berlin/Boston: De Gruyter Oldenbourg. In German. p. 46 (split-ballot experiments).

Malhotra, N. K. and Birks, D. F. (2007). Marketing Research. An Applied Approach (3rd European edition). Harlow: Pearson Education. p. 303 (causality), pp. 306-312 (experiment, validity, extraneous variables, randomization, true experimental designs), p. 314 (control group), p. 321 (factorial designs).

Marx, P. (2026). Principles of Survey Research. Online course, chapter 5, slide 209. QUESTIONSTAR. Slide 209.

Möhring, W. and Schlütz, D. (2019). Die Befragung in der Medien- und Kommunikationswissenschaft. Eine praxisorientierte Einführung [Surveys in media and communication studies: a practical introduction] (3rd ed.). Wiesbaden: Springer VS. In German. p. 19.

Schnell, R. (2019). Survey-Interviews. Methoden standardisierter Befragungen [Survey interviews: methods of standardized surveys] (2nd ed.). Wiesbaden: Springer VS. In German. DOI 10.1007/978-3-531-19901-6. pp. 54-56 (experiments, survey experiments), p. 135 (split-ballot in pretest and main survey).

Schwarz, N., Knäuper, B., Hippler, H. J., Noelle-Neumann, E. and Clark, L. (1991). Rating Scales: Numeric Values May Change the Meaning of Scale Labels. Public Opinion Quarterly, 55(4), 570-582. p. 572 (experiment 1). Also cited in Jacob et al. (2019), p. 46.

I wish you tests that settle a question, and the calm to report an inconclusive result as inconclusive.

Related articles:

• Survey incentives: what works online and what you gain
• P-value and statistical significance, explained simply
• Correlation vs. causation: how to read r and regression

Dr. Paul Marx

Author: Dr. Paul Marx
Dr. Paul Marx is the founder and Managing Director of QUESTIONSTAR and was a professor of marketing at the University of Siegen.