Top Ten Rules for Writing Surveys
From QualtricsWiki
[edit] Top 10 Rules for Writing Great Questions
Qualtrics Survey Software doesn't do everything for you when building great online surveys. Survey building is an art, and like all arts, it requires a great amount of work, practice, and help from others. The following points provide an overview of the common pitfalls we often see in bad questionnaires. Avoid these pitfalls and you are well on your way to understanding how to create a great survey questionnaire.
- Avoid loaded or leading words or questions. Subtle wording differences can produce great differences in results. "Could", "should", and "might" all sound about the same, but may produce a 20% difference in agreement to a question (The supreme court could.. should.. might.. change the limits on free speech in light of terrorist activities).
- Strong words, such as "prohibit" may represent control or action and influence your results (Do you believe that congress should prohibit insurance companies from raising rates?)
- Sometimes wording is just biased: You wouldn't want to go to Rudolpho's Restaurant for the company's annual party, would you?
- Misplaced questions. Questions placed out of order or out of context should be avoided. In general, a funnel approach is advised. Broad and general questions at the beginning of the questionnaire as a warm-up (What kind of restaurants do you most often go to?). Then more specific questions, followed by more general easy to answer questions (like demographics) at the end of the questionnaire.
- Mutually non-exclusive response categories. Multiple choice response categories should be mutually exclusive so that clear choices can be made. Non-exclusive answers frustrate the respondent and make interpretation difficult at best.
- Nonspecific questions. Build questions that are clearly understood.
- "Do you like orange juice?" This question is very unclear. Do I like what about orange juice? Taste, texture, nutritional content, Vitamin C, cancer prevention properties, the current price, concentrate, or fresh squeezed? Be specific about what you want to know. Issues of meaning and frequency are particularly difficult to specify: "Do you watch TV regularly?" What is regularly? Does it matter what I watch - is a DVD the same as TV?
- Confusing or unfamiliar words. Asking about caloric content, bits, bytes, and other industry specific jargon and acronyms is confusing. Make sure your audience understands your language level, terminology, and above all, that they understand what you are asking.
- Non-directed questions that give respondents excessive latitude.
- For example: "What suggestions do you have for improving tomato juice?" This question may be intended to obtain suggestions about improving taste, but the respondent may offer suggestions about texture, the type of can or bottle, about mixing juices, or even suggestions relating to use of tomato juice as a mixer or in recipes.
- Forcing respondents to answer. Respondents may not want or be able to provide the information requested. Privacy is an important issue to most people. Questions about income, occupation, finances, family life, personal hygiene, and personal, political or religious beliefs can be too intrusive and rejected by the respondent. Incentives and assurances of confidentiality do make it easier to obtain private information.
- Non-exhaustive listings. Do you have all of the options covered? If you are unsure, conduct a pretest using "Other (please specify)" as an option. When building multiple choice survey questions, make sure to cover at least 90% of the respondent answers.
- Unbalanced listings. Unbalanced scales may be appropriate for some situations and biased in others. When measuring alcohol consumption patterns, one study used a quantity scale that made the heavy drinker appear in the middle of the scale with the polar ends reflecting no consumption and an impossible amount to consume. Similarly, we expect all hotels to offer good care and may use a scale of excellent, very good, good, and fair. We do not expect poor care.
- Double barreled questions. Two questions should not be asked simultaneously. For example, "What is the fastest and most economical Internet service for you?" The fastest is certainly not the most economical.
