Avoiding the ‘Yes’ Bias
It’s often harder to say ‘no’ than ‘yes.’ This has been the subject of numerous self-help book chapters and blog posts. However, many people are surprised to learn that this is also a serious issue to consider in survey design.
Decades of research in survey methodology and psychology have shown that people generally tend to avoid saying ‘no.’ In the survey context this is called acquiescence response bias and it is a serious threat to data quality.
In the last post we wrote about using optimal scale labels to avoid biasing your questions. Acquiescence response bias is another thing you need to actively try to avoid. Some respondents, including those who are not highly motivated to think through the questions, take mental shortcuts when they are responding to questions. This tendency to answer positively is one of those common shortcuts.
But the problem doesn’t end there. The social norm of appearing agreeable and polite can bias your respondents, even when they are taking your survey anonymously. This makes them more likely to avoid disagreeing.
The effects of this bias can lead to harmful results—such as invalid conclusions and incorrect decisions being drawn from your data. A number of response formats enable acquiescence response bias including:
- Agree / disagree
- True / false
- Yes / no
Some have erroneously assumed that they can solve this problem by simply balancing their questionnaires. For example, making it so that in half the responses agreement would indicate a high level on the subject of the question and for the other half, agreement would indicate a low level. However, this approach has problems and should be avoided, because it assumes respondents are equally likely to acquiesce on all items. It also artificially pushes the aggregate responses toward the midpoint when there is no reason to believe they actually belong there.
Below is an example of a question type that is likely to produce acquiescence response bias:
To avoid the risk of acquiescence response bias, it would be better to simply ask how satisfied they feel about their experience. We have reformulated the question in a way that avoids asking for agreement below:
The same logic holds true for Yes / No and True / False response options. Rather than asking someone the following question:
You can use a response format that is less prone to the bias and also likely to provide more detailed data, like in the following example.
By using response formats and options that correspond more closely with the subject of your question, you also make interpretation easier for the respondents. While it may seem counterintuitive to some, this approach is actually more respondent-friendly and easier for respondents than the more biased approaches. This is particularly true when there are many questions that could potentially be formatted to use the same response scale.
In general, it is best to simply avoid using any response format that could enable acquiescence response bias.
May 21, 2020