Time is money, and often researchers want to send their surveys to respondents as quickly as possible, many times untested. But rushing to distribute a survey can often result in unforeseen problems with the data collection.

Using Qualtrics Survey Software can help you avoid errors in the first place but its still always a good idea to test surveys before you send them. Running your survey through a series of tests to check for potential problems can save you a lot of headaches down the road and ensure that you get the data that you want. In this post we discuss six different strategies for testing surveys before starting your data collection:

1. Respondent Debriefing

It is a good idea to run your survey on a small number of respondents prior to sending it out to your entire sample to get feedback on your survey. This is called “respondent debriefing.” For this approach, you will need to add several evaluation questions to the end of your survey for the respondents to answer. These can be open-ended or closed-ended questions and usually focus on assessing respondent comprehension and interpretation of survey questions. It should also include overall evaluations of the survey content, time, satisfaction and difficulty.

2. Cognitive Interviewing

“Cognitive interviews” are a good way to really understand what is going on the minds of your respondents when they are answering your questions. These are typically performed face-to-face with a small sample of 5–15 respondents. As the respondents answer each survey question, they are asked to “think aloud,” which can include paraphrasing, providing retrospective thinking or providing judgments of their confidence in what each question means.

3. Expert Evaluation

Your survey can be dramatically improved by feedback from two types of experts: 1) topic experts that have deep knowledge and expertise about the subject matter of your survey, and 2) survey methodologists that have expertise in how to collect the most accurate data for your research question. These expert evaluations can help shape the content and form of your survey and result in better data quality and more valuable insights.

4. Focus Groups

In the preliminary phases of questionnaire development, it can be very helpful to ask a focus group discuss your survey. These discussions, which are usually semi-structured discussions between 7–15 people led by a moderator, are particularly helpful for clarifying basic concepts in the survey and evaluating perceptions of respondent burden or topic sensitivity.

5. Experiments

Splitting a pre-test sample of respondents into groups and testing different variations of your survey design and content can be very powerful for understanding the results you will get when you field your main survey. These experiments are particularly useful for understanding how changes in question-wording, questionnaire design, visual layout, question order, and many other methodological factors may influence the data you collect.

6. Pilot Studies

Testing the final version of your survey on a small sample of your target population is critical—it can give you a sense of the kind of responses you will receive and any issues that may arise during the real survey period. Pilot studies often serve as a ‘dry run’ and are typically done just before fielding the survey to the entire sample. It is usually a good idea to include some evaluative questions, such as respondent perceptions of the length or difficulty of the questionnaire, satisfaction with taking the survey, etc. At Qualtrics we typically recommend that our customers use a sample of about 50 respondents for these pilot studies, or ‘soft launches,’ unless you need to do additional testing across different demographics.

7 Tips for Writing Great Survey Questions

Download PDF

This post was originally published in August 2015 and was republished in June 2018