Loading...
  • Customer Experience
    Customer Experience
  • Employee Experience
    Employee Experience
  • Brand Experience
    Brand Experience
  • Product Experience
    Product Experience
  • Core XM
    Core XM
  • Design XM
    Design XM

Choice Randomization

What's on This Page:


Was this helpful?


This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

The feedback you submit here is used only to help improve this page.

That’s great! Thank you for your feedback!

Thank you for your feedback!


About Choice Randomization

Randomization is an important research technique used to help overcome the bias that can result from the order items are presented in. With Qualtrics, you can randomize choices, questions, or entire blocks of questions. This page outlines how to randomize choices.

Setting Up Choice Randomization

There are five types of choice randomization:

  • No Randomization: The default option. Choices are not randomized.
  • Display choices in a random order: Randomizes all choices.
  • Present only __ of total choices: Randomly displays a specified number of choices from the total choices available.
  • Randomly flip the order of choices: Randomly reverses the order of presented choices.
    • Consistently reverse the choice order for all questions with this randomization option selected: This option only appears after enabling “randomly flip the order of choices.” When enabled, this option will consistently randomize answer choice order for your survey.
  • Advanced Randomization: Specify which choices are randomized and which are always displayed in the same location.

To set up choice randomization

  1. While editing your survey, select the question you want to randomize.
    selecting a question in the editor and clicking choice randomization
  2. Select Choice randomization.
    Qtip: For matrix table questions, there will be two options: Statement randomization and Scale point randomization.
  3. Select the type of randomization to apply.
    choosing a randomization option and clicking Save
  4. Click Save.

Once you’ve applied randomization, a randomization icon will appear in the upper right corner of the question. Clicking on this icon will open the choice randomization window.

a question with the two crossing arrow symbol, denoting randomization

Advanced Randomization

Some questions require a more complex setup than simply randomizing all choices or displaying a random subset of choices. For example, all choices in the following question should be randomized, except “Other”, which should remain at the bottom of the list.

a multiple choice question with a list of options. other is at the bottom of the list

Advanced randomization can help with these types of scenarios by dividing choices into the five categories below.

Fixed Display Order

Choices in this category will always display in the same position in the choice list. The up and down arrows to the left can be used to rearrange the list order.

All choices in the Fixed Display Order box have been moved to the Randomize Choices box on the right except Other and Product Photos, which we want to always be last and first, respectively

For this question, “Product Photos” will always show at the top of the list, “Other” will always show at the bottom, and all other choices will be inserted randomly in the middle.

Randomize Choices

Choices in this category will be inserted randomly into the question in the spots labeled {Randomized}.

Other is last in the list of Fixed Display Order. All moved choices are represented by brackets around the word Randomized

For this question, all choices are randomized, except “Other”, which will always display at the bottom.

Random Subset

Choices in this category will be inserted randomly into the question in the spots labeled {Random Subset}. With this category, you specify the number of choices to be inserted and whether the choices should be evenly presented or completely randomized.

Four choices are in Random Subset to the right. The settings say randomly insert 2 questions from this list, so on the left, it says Random Subset TWICE where the items should be randomized

For this question, one of the three Random Subset options will show first in the list and another will show third. The remaining subset option will not be displayed.

Randomly Reverse Choices

Choices in this category will be randomly displayed in either their original order, or in a reverse order.

the randomly reverse choices option in advanced randomization

You can also choose to have your answer choices consistently reversed across all questions by enabling the Consistently reverse the choice order for all questions with this randomization option selected setting.

Excluded Items

Choices in this category will never be used.

the excluded items section of advanced randomization

Qtip: Why have a choice that will never be displayed? This can happen when using the carry forward feature. With carry forward, the choices are brought in from a previous question. Though a choice like “None of the above” may make sense in the original question, it may not be needed in the follow up question. Additionally, it is beneficial to hide questions and answer choices instead of deleting them in order to avoid unintentionally removing collected data from your data set.

To set up advanced randomization

  1. While editing your survey, select the question you want to add advanced randomization to.
    selecting a question in the editor and clicking choice randomization
  2. Click Choice randomization.
  3. Select Advanced Randomization. The advanced randomization window will then open automatically; if it does not, click Set Up Advanced Randomization.
    selecting the advanced randomization option
  4. Select the items to be randomized. Hold down Shift or Ctrl while choosing items to select multiple at a time.
    selecting the choices to randomize, which are highlighted in blue
  5. Use the arrows to move the choices into one of the four categories (Randomize choices, Random subset, Randomly reverse choices, or Excluded items).
    clicking the arrow buttons to move the selected choices to a randomize group, and clicking save
  6. When finished, click Save.

Preserving Randomization Order

In some studies, multiple questions ask about the same set of choices. The random order used to present choices in the first question may need to be preserved so the same order is used in subsequent questions.

Randomization order can be preserved from one question to the next by using the carry forward feature.

To preserve randomization order

  1. Set up randomization on the initial question using the steps described above.
    after adding randomization to the question, clicking add new question
  2. Click Add new question and create a follow up question without any choices.
  3. Select your newly added question. clicking the follow-up question and selecting carry forward choices
  4. Click Carry forward choices.
  5. Select the question with choice randomization.
    creating carry forward logic to use the selected choices from the randomized question
  6. Choose Selected Choices.
  7. Click Confirm.

Randomization Data

You can view the order in which your choices were displayed to respondents by enabling Export Randomized Viewing Order before exporting your data. The display order will appear as a column in your data file with multiple choices separated by a bar “|” if multiple choices were seen.

CSV with randomized data. The column heading has DO-Q-then the question number.

If you export with Split multi-value fields into columns selected, too, this will separate the viewing order into separate columns, instead of listing all displayed choices underneath one column.

To download randomization data

  1. Navigate to the Data & Analysis tab.
    navigating to data & analysis, clicking export & import, and selecting export data
  2. Click Export & Import.
  3. Click Export Data.
  4. Choose the file type you would like to download.
    Bar along the top allows file type selection; More Options on bottom-left
  5. Click the blue colored More Options text.
  6. Select the Export viewing order data for randomized surveys checkbox.
    Exporting viewing order data towards the bottom of list of choices; Download button in bottom-right
  7. Click the green Download button.
Qtip: To learn more about exporting data, view our Download Options page.