How to Avoid Bad Survey Responses
Over the past few weeks, we’ve talked about getting in your respondent’s heads, the threats posed by acquiescence bias, ‘don’t know’ responses, straightlining, and selecting the first acceptable response option. Taken together these threats form the behaviors predicted and described by the theory of survey satisficing. That’s a fancy way of saying that respondents just try to meet the lowest threshold of acceptability for an answer, rather than making the time to give the best response, and that hurts your data quality.
Knowing the problem is out there begs the question, how do you protect against it?
First, there are two types of survey satisficing – weak and strong. Weak satisficing happens when a respondent tries to shortcut the cognitive response process, by not fully engaging with your survey questions. For example, they don’t search their memory as thoroughly as we would like or they don’t integrate the information they’ve retrieved from their memories very well.
Strong satisficing, on the other hand, is much more flagrant. This happens with respondents do not attempt to engage with the question at all, but rather revert to bad behaviors such as primacy or selecting the first plausible response they encounter.
What’s the solution? Our past several posts have described how you can design your surveys to make it more difficult for respondents to engage in satisficing. These mitigating steps include:
- Not using scales that push respondents to agree
- Avoiding ‘don’t know’ response options
- Getting rid of grid questions
Randomizing the order of list response options
In the coming weeks, we will discuss what causes satisficing and what you can do to actively combat these data-degrading respondent behaviors by increasing respondent motivation and reducing survey difficulty.
Satisficing in all of its forms is a huge threat to the reliability and validity of survey data. Making smart design decisions will help reduce the likelihood that your respondents will engage in satisficing, leading to higher-quality data, less distortion, and more reliable conclusions.