6 Ways to Pretest Your Survey Before You Send It
Time is money, and often researchers want to send their surveys to respondents as quickly as possible. But rushing to distribute a survey can often result in unforeseen problems with the data collection.
Running your survey through a series of tests to check for potential problems can save you a lot of headaches down the road and ensure that you get the data that you want. In this post we discuss six different strategies for testing surveys before starting your data collection:
- Respondent debriefing
It is a good idea to run your survey on a small number of respondents prior to sending it out to your entire sample to get feedback on your survey. This is called “respondent debriefing.” For this approach, you will need to add several evaluation questions to the end of your survey for the respondents to answer. These can be open-ended or closed-ended questions and usually focus on assessing respondent comprehension and interpretation of survey questions. It should also include overall evaluations of the survey content, time, satisfaction and difficulty.
- Cognitive interviewing
“Cognitive interviews” are a good way to really understand what is going on the minds of your respondents when they are answering your questions. These are typically performed face-to-face with a small sample of 5–15 respondents. As the respondents answer each survey question, they are asked to “think aloud,” which can include paraphrasing, providing retrospective thinking or providing judgments of their confidence in what each question means.
- Expert evaluation
Your survey can be dramatically improved by feedback from two types of experts: 1) topic experts that have deep knowledge and expertise about the subject matter of your survey, and 2) survey methodologists that have expertise in how to collect the most accurate data for your research question. These expert evaluations can help shape the content and form of your survey and result in better data quality and more valuable insights.
- Focus groups
In the preliminary phases of questionnaire development, it can be very helpful to ask a focus group discuss your survey. These discussions, which are usually semi-structured discussions between 7–15 people led by a moderator, are particularly helpful for clarifying basic concepts in the survey and evaluating perceptions of respondent burden or topic sensitivity.
Splitting a pre-test sample of respondents into groups and testing different variations of your survey design and content can be very powerful for understanding the results you will get when you field your main survey. These experiments are particularly useful for understanding how changes in question wording, questionnaire design, visual layout, question order, and many other methodological factors may influence the data you collect.
- Pilot studies
Testing the final version of your survey on a small sample of your target population is critical—it can give you a sense of the kind of responses you will receive and any issues that may arise during the real survey period. Pilot studies often serve as a ‘dry run’ and are typically done just before fielding the survey to the entire sample. It is usually a good idea to include some evaluative questions, such as respondent perceptions of the length or difficulty of the questionnaire, satisfaction with taking the survey, etc. At Qualtrics we typically recommend that our customers use a sample of about 50 respondents for these pilot studies, or ‘soft launches,’ unless you need to do additional testing across different demographics.