Reducing Bias In Survey Responses: The Role of Randomization
When presented with a list of options to choose from, many survey respondents will select the first reasonable alternative and then move on, rather than reading all of the options and selecting the most appropriate answer(s) for the question. Selecting the first reasonable option is one of the strategies employed by respondents that are not engaging in the optimal process of responding to a question that we discussed on this blog a few weeks ago. Survey designers often fail to recognize the bias that this can introduce into their data. This bias can be hard to detect in the data, which makes the problem difficult to solve after data collection has completed.
This tendency, known as ‘primacy,’ is common in online and paper surveys. Another similar tendency, common in telephone and face-to-face surveys, is called ‘recency’ – where the respondent selects the most recently mentioned acceptable response option presented by the interviewer.
Primacy and recency can produce biases in your data that are hard to detect. Fortunately, there is a relatively simple solution: randomizing the order of the list presented to each respondent. While this doesn’t guarantee your respondents will engage with the question, it generally turns what would have been a bias in your data into variance.
Qualtrics makes randomizing your response options easy. You can also randomize a subset of the items so that a ‘None of these’ option always appears last in the list:
It is important to remember not to randomize response scales, which respondents expect to follow a coherent order with the maximum value on one end and the minimum value on the other. You can, however, randomize the direction of the response options while still maintaining a sensible order:
Even though randomization can help remove biases in your data, it doesn’t solve the lack of engagement by your respondents. There is not a solution for every poor respondent tendency, but being aware of behaviors like primacy and recency can help you design better surveys and improve the overall quality of your data.