A/B Tests

About A/B tests

A/B tests are a proven method used across various disciplines to test differences in how survey respondents react to two or more versions of a stimulus. A stimulus could be, for example, a question, an image, an advertisement, a piece of wording, the design of a web page, a video and much more. The goal of an A/B test is to determine the best version.

Survey respondents are randomly assigned to a group in which only one version of the stimulus is shown to them. They don't know which other versions are available. This ensures that respondents' assessments aren't influenced by comparison with other versions, so the results stay unbiased and objective.

In this article we'll show you how to set up and run A/B tests in QUESTIONSTAR.

Setting up or editing an A/B test

Requirements:To set up an A/B test, first create a questionnaire that contains all the versions of your stimulus or stimuli. These versions can either be placed on one questionnaire page or, if you prefer, given a separate page for each version. Over the course of the survey, each respondent is shown only one version of the stimulus, while all the other versions stay hidden.

To activate the A/B test or open it for editing:

1. Click the three dots in the top menu bar.
2. Select the item "AB Tests".

a) If the versions of your stimulus are within a single questionnaire page or spread across several pages (but aren't complete pages):

3. Switch to the "Questions" tab.
4. Select the questions that belong to one scenario.
5. Click the button with the right-pointing arrow next to the box of the relevant test version, or drag one of the questions with the mouse into the version field to assign the selected questions to the test version. Alternatively, you can move the questions into the version field one at a time.

b) If your test versions are complete pages, proceed the same way as described above, but select "Pages" in step 3.

In this case too, several pages that don't necessarily follow one another can be assigned to a test version.

A/B/C/D/…: tests with more than two versions

Often you want to test more than just two versions of a stimulus. QUESTIONSTAR sets no limits in this respect and lets you test as many versions as you like.

1. You can add a test version by clicking the "Add Test Version" button.
2. To delete an incorrectly created or unnecessary test version, click the trash icon at the top right of the version field.

Multiple A/B tests in one questionnaire

Sometimes you want to test several stimuli that are independent of one another within one questionnaire. You can do this with QUESTIONSTAR too. To that end, we provide you with the following functions:

1. To add a new test with an additional stimulus, click the plus icon next to the test tab (2) and assign questions or pages to the new test. You can test up to four different stimuli (separate A/B tests) in one questionnaire in total.
2. The test tab lets you switch between the different A/B tests. Just click a test in the tab to switch to its settings.
3. You can delete a test you no longer need by selecting it in the test tab and then clicking the trash icon.

How A/B tests are marked in the questionnaire

In QUESTIONSTAR you can easily see which pages and questions are part of an A/B test.

1. In the left navigation panel: Pages and questions that are part of an A/B test are highlighted in green in the left navigation panel. This highlighting lets you grasp at a glance which pages and questions are assigned to which scenarios of an A/B test.
2. In the editor area: In the central editor area, every question or page that belongs to an A/B test is also marked with a special "A/B" icon. Right next to this icon is the letter of the corresponding test scenario (e.g. "A" or "B"). This way you can clearly tell which scenario the displayed question is assigned to.

Thanks to these clear markings, you keep an overview of your A/B tests and the scenarios they contain at all times.

How are survey respondents assigned to the test versions?

At the core of A/B tests is the idea that survey respondents are randomly assigned to the different test versions. At the same time, the aim is for the respondent groups assigned to the versions to be filled roughly evenly, in order to ensure the groups are comparable — particularly with regard to the number of members. A purely random assignment of respondents to the groups can, however, result in the distribution of respondents across the groups not being entirely even.

To counteract this trade-off and ensure an even filling of the version groups, QUESTIONSTAR uses the following approach:

When a new respondent opens the survey and reaches the section where the A/B test begins, the QUESTIONSTAR engine checks the fill state of the A/B tests and selects the test version that has the fewest completed questionnaires. The respondent is assigned to that version. This ensures both a (roughly) even filling and a random assignment of the versions. (The moment at which the respondent opens the survey is random from the survey creator's point of view; so the fill state of the different test versions at that moment is likewise random. This satisfies both requirements — the random assignment to the test versions and the even filling of the test scenarios.)

What happens if a survey contains several A/B/C/… tests?

When a survey contains several A/B tests, QUESTIONSTAR creates what is known as a full factorial design plan. All possible combinations of the versions of the different tests are created and the experimental groups assigned accordingly.

When a new respondent opens the survey, it is checked which of the groups is filled the least. The respondent is then assigned to that group.

Why aren't the groups filled exactly evenly anyway?

Especially with surveys sent to many (hundreds or thousands of) test persons, it can happen that several respondents fill in the survey at the same time. However, in deciding which group is filled the least, only those responses are counted that are already finished — and therefore complete — at the moment the respondent reaches the point in the survey where the A/B test begins.

It can happen, though, that respondents abandon the survey and are therefore not included in the evaluation. Whether this is the case cannot be known at the moment of group assignment. That's why it can happen that several survey respondents are assigned to the same group at that moment. This can lead to slight deviations in the number of respondents assigned to the different test groups.

Question search and fill state

Two further functions not covered in the guide so far, which can certainly prove helpful when working on setting up A/B tests, are the question search and the fill state:

1. Question search: Especially with longer questionnaires, finding the right question isn't always easy. But if you enter part of the question or page label, or its number, in the search field, you can find it more quickly.
2. Fill state: Once the survey has started and responses have come in, the fill state indicator is displayed.
3. Fill state report: You can open a detailed report on the fill state in a new browser window by clicking the button (3). You can then share the link from the browser's address bar with your team, your superiors or your clients.

Correcting errors in the A/B test settings

Nobody's perfect, mistakes happen — that's not a problem! If you made a mistake while setting up your A/B test, you can correct it at any time.

1. To remove an element (question or page) that was mistakenly added to a test version, select the element and click the button with the left-pointing arrow, or move the element to the left — into the list with the questionnaire structure.
2. You can delete an unnecessary test version (along with all its contents) by clicking the button with the trash icon above the version you no longer want. Please note, however, that an A/B test must contain at least two versions — otherwise the test makes no sense.
3. If you unintentionally deleted a version of the test, you can add a new version at any time by clicking the "Add Test Version" button.
4. You can delete an unnecessary test by selecting the relevant test in the tests tab and then clicking the trash icon in the active tab.

Demo questionnaire to copy into your own account

Try out how A/B tests work in QUESTIONSTAR live. To do so, open 🔗 this link.