Survey Speeding, Part 2: Designing Surveys to Avoid Harm
Previously we highlighted some of the concerns surrounding people that appear to respond very quickly to surveys. While concerns about survey speeding are valid, there are indications that the solution is not to simply throw out these respondents from your dataset.
Despite the evidence that speeders don’t really appear to be doing that much harm, you should take steps to encourage thoughtful responses to your survey questions whenever possible. That diligence may sometimes translate into slower completion times for your questions and surveys.
Tip 1: Measure response times and look at subsets
A good place to start is to measure response times to every question in your survey and assess which individual questions you think respondents might not be answering thoughtfully. Once you have this timing data, analyze the full dataset as you normally would. Afterward, compare the subset of responses that you think may be too fast with the subset you think is slow enough. Chances are you won’t find any substantial biases.
Tip 2: Look at question design
If you find questions that you feel are commonly being answered too quickly you can assess the design features of those questions to see if there are improvements you can make. Consider the potential effect respondent fatigue is having on your survey, as it’s a likely cause of fast responses given late in the survey or to especially difficult or complex questions.
Applying best practices for survey design, such as not using matrix/grid question types, using construct-specific response options, and expressing gratitude to your respondents can help encourage more thoughtful responding. Many of the other recommendations made in this blog series and the broader survey methodology literature can help, too.