A/B Testing in Surveys

About A/B Testing in Surveys

A/B Testing is a classic research format used across disciplines. In it, you compare a control condition to one where a treatment is applied, so you can then analyze if there is a significant difference between the respondents’ behavior in the treatment vs. the control baseline. A/B Testing can also be useful in web development and marketing because it allows you to compare two versions of a product or design in order to determine which performs better, or is preferred by your customers.

A/B Testing can be performed in Qualtrics by randomly assigning respondents to one of the two (or more) conditions, without letting the respondents know what the other condition looks like. This allows respondents to provide opinions unbiased by the desire to compare and contrast options.

Basic Setup

This section goes over the easiest way to set up A/B Testing in Qualtrics. This setup is for basic studies that don’t require multiple blocks or other elements within each condition.

1. Start by making your introductory questions in one block.
   
   Qtip: This first block often serves as a consent form, demographic questions, or other general content that every respondent should see, regardless of condition.
2. Click Add Block and create a block for each condition your study has.
3. Build each unique condition as its own block.
   
4. Navigate to the Survey flow.
5. Click Add a New Element Here.
   
6. Select Randomizer.
7. Move the condition blocks under the Randomizer, so they are nested below the Randomizer.
   
8. Set how many blocks you would like to randomly present. If you’re assigning each respondent just one condition, then type 1.
   
9. Make sure Evenly Present Elements is selected. This ensures your conditions are evenly distributed across your sample.
10. Click Apply.
    Qtip: You can add additional blocks to your survey that are shared by all respondents, regardless of condition. Just make sure that in the survey flow, blocks you don’t want randomly presented are not indented beneath the Randomizer. (See the Follow-Up Questions block in the above example.)

Setup for Complex Conditions

The steps discussed in this section allow you to add multiple blocks or elements into each condition. The branches themselves will be randomized, but once a branch is assigned, the condition will proceed through all the contents of the branch in order.

In addition, randomization data will not export the name of a block by default, instead returning a flow ID. In order to get more clearly labelled randomization data, we will set up certain parts of the Survey Flow differently than in the basic A/B setup. Following the example taught in the steps below, all randomized block data will be under the Condition column of your data.

1. Start by making your introductory questions in one block.
   
   Qtip: This first block often serves as a consent form, demographic questions, or other general content that every respondent should see, regardless of condition.
2. Click Add Block and create a block for each module your study has.
3. Build your survey contents. Be sure to keep content from different conditions in separate blocks.
   
4. Navigate to the Survey flow.
5. Click Add a New Element Here.
   
6. Select Randomizer.
7. Underneath the randomizer, click Add a New Element Here and select Embedded Data.
   
8. Name the embedded data Condition and set it equal to the name of the first condition. In this example, we’re calling it Old Logo.
   
9. Underneath the randomizer, click Add a New Element Here and select Embedded Data again.
10. Name the embedded data Condition and set it equal to the name of the condition. This must be different from Step 8. In this example, we’re calling it New Logo.
    
11. Set how many blocks you would like to randomly present. If you’re assigning each respondent just one condition, then type 1.
12. Make sure Evenly Present Elements is selected. This ensures your conditions are evenly distributed across your sample.
13. In your survey flow, click Add a New Element Here and select Branch.
    Qtip: This branch must be under the randomizer, but it must not be indented under the Randomizer! Use the images in these Steps as reference.
14. Set the condition of the branch so that Embedded Data named Condition must be Equal to Old Logo.
    
15. Drag the elements and blocks from the Old Logo condition so they are nested under the branch.
    
16. In your survey flow, click Add a New Element Here and select Branch.
    Qtip: This branch must be under the randomizer, but it must not be indented under the Randomizer or the other Branch! Use the images in these Steps as reference.
17. Set the condition of the branch so that Embedded Data named Condition must be Equal to New Logo.
    
18. Drag the elements and blocks from the New Logo condition so they are nested under the branch.
    
19. Click Apply.
    Qtip: You can add additional blocks to your survey that are shared by all respondents, regardless of condition. Just make sure that in the survey flow, blocks you don’t want randomly presented are not indented beneath the randomizer or the branches you’ve created.

Special thanks to Mary Curry, winner of the Community Competition for “Best Common Use Case Page.”