Loading...
  • Customer Experience
    Customer Experience
  • Employee Experience
    Employee Experience
  • Brand Experience
    Brand Experience
  • Product Experience
    Product Experience
  • Core XM
    Core XM
  • Design XM
    Design XM

A/B Testing in Surveys

What's on This Page:


Was this helpful?


This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

The feedback you submit here is used only to help improve this page.

That’s great! Thank you for your feedback!

Thank you for your feedback!


About A/B Testing in Surveys

A/B Testing is a classic research format used across disciplines. In it, you compare a control condition to one where a treatment is applied, so you can then analyze if there is a significant difference between the respondents’ behavior in the treatment vs. the control baseline. A/B Testing can also be useful in web development and marketing because it allows you to compare two versions of a product or design in order to determine which performs better, or is preferred by your customers.

A/B Testing can be performed in Qualtrics by randomly assigning respondents to one of the two (or more) conditions, without letting the respondents know what the other condition looks like. This allows respondents to provide opinions unbiased by the desire to compare and contrast options.

Basic Setup

This section goes over the easiest way to set up A/B Testing in Qualtrics. This setup is for basic studies that don’t require multiple blocks or other elements within each condition.

Qtip: This example has just two conditions, but this format can be used to build as many conditions as you want.
Qtip: See Randomization Data for how to export and interpret the results. By default, randomization data will not export the name of a block, but the Flow ID. If you’d like your condition data to have clearer labelling, see the next section.
  1. Start by making your introductory questions in one block.
    Survey with introductory block with one descriptive text in it. Add Block link in blue below this block

    Qtip: This first block often serves as a consent form, demographic questions, or other general content that every respondent should see, regardless of condition.
  2. Click Add Block and create a block for each condition your study has.
  3. Build each unique condition as its own block.
    Condition 1 and Condition 2 blocks
  4. Navigate to the Survey Flow.
  5. Click Add a New Element Here.
    Inside survey flow adding randomizer at bottom. Randomizer choice is pink, top yellow row of choices
  6. Select Randomizer.
  7. Move the condition blocks under the Randomizer, so they are nested below the Randomizer.
    Moving blocks with blue move link to be nested under randomizer at bottom
  8. Set how many blocks you would like to randomly present. If you’re assigning each respondent just one condition, then type 1.
    Finished survey flow
  9. Make sure Evenly Present Elements is selected. This ensures your conditions are evenly distributed across your sample.
  10. Click Save.
    Qtip: You can add additional blocks to your survey that are shared by all respondents, regardless of condition. Just make sure that in the Survey Flow, blocks you don’t want randomly presented are not indented beneath the Randomizer. (See the Follow-Up Questions block in the above example.)

Setup for Complex Conditions

The steps discussed in this section allow you to add multiple blocks or elements into each condition. The branches themselves will be randomized, but once a branch is assigned, the condition will proceed through all the contents of the branch in order.

In addition, randomization data will not export the name of a block by default, instead returning a Flow ID. In order to get more clearly labelled randomization data, we will set up certain parts of the Survey Flow differently than in the basic A/B setup. Following the example taught in the steps below, all randomized block data will be under the Condition column of your data.

Qtip: This example has just two conditions, but this format can be used to build as many conditions as you want.
  1. Start by making your introductory questions in one block.
    Survey with introductory block with one descriptive text in it. Add Block link in blue below this block

    Qtip: This first block often serves as a consent form, demographic questions, or other general content that every respondent should see, regardless of condition.
  2. Click Add Block and create a block for each module your study has.
  3. Build your survey contents. Be sure to keep content from different conditions in separate blocks.
    Condition 1 and Condition 2 blocks
  4. Navigate to the Survey Flow.
  5. Click Add a New Element Here.
    Inside survey flow adding randomizer at bottom. Randomizer choice is pink, top yellow row of choices
  6. Select Randomizer.
  7. Underneath the Randomizer, click Add a New Element Here and select Embedded Data.
    Selecting green Embedded Data Element
  8. Name the Embedded Data Condition and set it equal to the name of the first condition. In this example, we’re calling it Old Logo.
    Adding Embedded Data
  9. Underneath the Randomizer, click Add a New Element Here and select Embedded Data again.
  10. Name the Embedded Data Condition and set it equal to the name of the condition. This must be different from Step 8. In this example, we’re calling it New Logo.
    Refining randomizer up top and then going back to the bottom to add a new element
  11. Set how many blocks you would like to randomly present. If you’re assigning each respondent just one condition, then type 1.
  12. Make sure Evenly Present Elements is selected. This ensures your conditions are evenly distributed across your sample.
  13. In your Survey Flow, click Add a New Element Here and select Branch.
    Qtip: This branch must be under the Randomizer, but it must not be indented under the Randomizer! Use the images in these Steps as reference.
  14. Set the condition of the Branch so that Embedded Data named Condition must be Equal to Old Logo.
    Branch with described condition set
  15. Drag the elements and blocks from the Old Logo condition so they are nested under the branch.
    Blocks dragged under previous branch, much below that creating a new branch
  16. In your Survey Flow, click Add a New Element Here and select Branch.
    Qtip: This branch must be under the Randomizer, but it must not be indented under the Randomizer or the other Branch! Use the images in these Steps as reference.
  17. Set the condition of the Branch so that Embedded Data named Condition must be Equal to New Logo.
    Branch with aforementioned condition set
  18. Drag the elements and blocks from the New Logo condition so they are nested under the branch.
    Save in green in bottom-right of window
  19. Click Save.
    Qtip: You can add additional blocks to your survey that are shared by all respondents, regardless of condition. Just make sure that in the Survey Flow, blocks you don’t want randomly presented are not indented beneath the Randomizer or the branches you’ve created.

Example: This is what the Survey Flow we used in this example looks like when it’s finished being built.

Introduction block, randomizer with two embedded data elements underneath, branch with two blocks underneath, another branch with two blocks underneath

Special thanks to Mary Curry, winner of the Community Competition for “Best Common Use Case Page.”