Getting In Your Respondents’ Heads
When you’re writing a survey, don’t forgot to think about the person who will actually be taking it. All too often, survey designers become so focused on what data they want to collect that they fail to consider the respondent perspective, and that can lead to problems.
When we write surveys, many of us idealize the process. We think it goes something like this:
- Respondents closely read and interpret the meaning of the question
- Respondents carefully search their memory for all relevant information about the subject of the question.
- Respondents make summary judgments about the information they just retrieved from memory.
- Respondents map the judgments about the subject of the question onto the response options and format provided for in the question.
Sounds amazing, right? Unfortunately, this is an idealized process and while we hope every question is answered that carefully, the reality is that people are busy, have things to do, and have limited attention spans.
Many respondents end up taking shortcuts around this optimal four-step response process. These shortcuts lead to less valid and reliable data. So it’s up to us to design surveys that maximize the chances that respondents are engaging in the optimal process. This will help data quality and should have the effect of leaving your respondents feeling happier about the survey (and more likely to respond again next time).
So what should you do? Well, for one, research indicates that respondents are most likely to provide their best data on questions that are simple and brief. In addition, there are a few helpful steps that any researcher can take to improve the chances that their respondents will thoughtfully and reliably answer the questions.
The first thing is to just be aware. Think about your respondents and this optimal response process when you are designing your survey. Would you want to take this survey? Are there too many questions? Some that are too complicated? By keeping the respondent experience at the forefront, you’ll reap dividends in data quality.
Second, take your own survey very carefully and slowly yourself, in the manner you would want your respondents to take it. Using this approach, you can identify complex, long, confusing, or unnecessary questions and either modify or delete them. It can be a big red flag you find yourself feeling impatient, bored, or wanting to answer questions superficially rather than thinking about the responses carefully. You probably care about this survey and your data far more than any of your respondents, so make sure you can tolerate it yourself.
Lastly, pre-test your survey and asking those initial respondents for their feedback. This can provide great insights into what’s going well and what isn’t. This can be especially helpful in terms of understanding how respondents are experiencing the survey questions and what factors may be interfering with the optimal response process.
Many researchers find in pre-testing that questions they thought were perfectly clear are confusing, or that their survey is far too long, or any number of other important things that can harm data quality. This pre-testing can easily be done first on colleagues, friends, or family members, and then on a small sample of the population of interest. It is critical to take this process seriously and not disregard the feedback, even if it means going back to the drawing board about the survey. In general this pre-testing should be done before and after any major changes to survey questions or questionnaires.