Survey Platform - Choice Randomization | Qualtrics

Choice Randomization

Introduction

Randomization is an important research technique used to help overcome the bias that can result from the order items are presented in. With Qualtrics, you can randomize choices, questions, or entire blocks of questions. This page outlines how to randomize choices.

Setting Up Choice Randomization

There are four types of choice randomization:

  • No Randomization: The default option. Choices are not randomized.
  • Randomize the order of all choices: Randomizes all choices.
  • Present only __ of total choices: Randomly displays a specified number of choices from the total choices available.
  • Advanced Randomization: Specify which choices are randomized and which are always displayed in the same location.

To set up choice randomization

  1. In the Survey module, select the question you want to randomize, then click the gray gear to access the Question Options and choose Randomization.image01
  2. Select a randomization type.image15
  3. Click Save when finished.

Once you’ve applied randomization, a randomization icon will appear to the left of the question. Clicking on this icon will open the Choice Randomization window.

image11

Advanced Randomization

Some questions require a more complex setup than simply randomizing all choices or displaying a random subset of choices. For example, all choices in the following question should be randomized, except “Other”, which should remain at the bottom of the list.

image02

Advanced Randomization can help with these types of scenarios by dividing choices into the four categories below.

Fixed Display Order

Choices in this category will always display in the same position in the choice list. The up and down arrows to the left can be used to rearrange the list order.

image20
For this question, “Product Photos” will always show at the top of the list, “Other” will always show at the bottom, and all other choices will be inserted randomly in the middle.

Randomize Choices

Choices in this category will be inserted randomly into the question in the spots labeled {Randomized}.

image00
For this question, all choices are randomized, except “Other”, which will always display at the bottom.

Random Subset

Choices in this category will be inserted randomly into the question in the spots labeled {Random Subset}. With this category, you specify the number of choices to be inserted and whether the choices should be evenly presented or completely randomized.

image21
For this question, one of the three Random Subset options will show first in the list and another will show third. The remaining subset option will not be displayed.

Undisplayed Items

Choices in this category will never be used.

image18
For this question, the option “Product Photos” will never be displayed in the question.
Qtip: Why have a choice that will never be displayed? This can happen when using the Carry Forward feature. With Carry Forward, the choices are brought in from a previous question. Though a choice like “None of the above” may make sense in the original question, it may not be needed in the follow up question.

To set up advanced randomization

  1. In the Survey module, select the question to be randomized, then click the gray gear to access the Question Options and choose Randomization.image01
  2. Click Set Up Advanced Randomization.image12
  3. Select the items to be randomized. Hold down Shift or Ctrl while choosing items to select multiple at a time.image14
  4. Use the arrows to move the choices into one of the three categories (Randomize Choices, Random Subset, or Undisplayed Items) then click Save.image03

Preserving Randomization Order

In some studies, multiple questions ask about the same set of choices. The random order used to present choices in the first question may need to be preserved so the same order is used in subsequent questions.

Randomization order can be preserved from one question to the next by using the Carry Forward feature.

To preserve randomization order

  1. Set up randomization on the initial question using the steps described above.image13
  2. Create the follow up question, without any choices.image09
  3. Click the gray gear to access the Question Options and select Carry Forward Choices.image04
  4. Select which choices to bring into the follow up question.image19
  5. Click Done.

Randomization Data

You can view the order in which your choices were displayed to respondents by enabling Export Randomized Viewing Order before exporting your data. The display order will appear in the last column of your data file and will be separated by a bar “|” if multiple choices were seen.

image08

To download randomization data

  1. Navigate to the Data & Analysis module and click on the Data tab.image17
  2. Click Export Data.image05
  3. Choose the file type you would like to download.image22
  4. Click the blue colored More Options text.image07
  5. Select the Export viewing order data for randomized surveys checkbox.image10
  6. Click the green Download button.image16
Qtip: To learn more about exporting data, view our Download Options page.