The 10 commandments for writing good surveys and questions
In the dark days of survey creation, survey question writing was confusing.
Then came forth the 10 commandments for writing good survey questions to guide everyone from elite researchers to entry level interns in all things survey question writing. A good survey questionnaire is made (or not made) by the individual questions that constitute it. The best survey theme, logic, and routing is meaningless without great survey questions.
Bind them to thy mind, let them flow through thy survey.
1. Thou shalt avoid loaded questions or leading words
Subtle wording differences can produce great differences in results.
"Could", "should", and "might" all sound about the same, but may produce a 20% difference in agreement to a question (the supreme court could /should/ might change the limits on free speech in light of terrorist activities).
Strong words, such as "prohibit" may represent control or action and influence your results.
If thou desire is clean data, this commandment must be adhered to.
2. Thou shalt honor the ordering of questions
Survey questions placed out of order or out of context should almost always be avoided.
In general, a funnel approach is advised. Broad and general questions at the beginning of the questionnaire as a warm-up. Then more specific questions, followed by more general easy to answer questions (like demographics) at the end of the questionnaire.
3. Thou shalt avoid non-specific questions
Build questions that are clearly understood.
Do you like orange juice? This is very unclear...do I like what about orange juice? Taste, texture, nutritional content, vitamin C, cancer prevention properties, the current price, concentrate, or fresh squeezed?
Be specific in what you want to know.
Issues of meaning and frequency are particularly difficult to specify: Do you watch TV regularly? (What is regularly? Does it matter what I watch? Is a DVD the same as TV?).
4. Thy question wording shall not be confusing or unfamiliar
Asking about caloric content, bits, bytes and other industry-specific jargon and acronyms is confusing.
Thy respondent must understand thy language level so that thy survey response rates may be high and data clean.
5. Thou shalt not force respondents to answer
Respondents may not want, or may not be able to provide the information requested.
Privacy is an important issue to most people.
Questions about income, occupation, finances, family life, personal hygiene and personal, political or religious beliefs can be too intrusive and rejected by the respondent.
Incentives and assurances of confidentiality will help in addition to adding a "prefer not to answer" option.
6. Thou shalt not adulterate your survey with non-exhaustive listings
Do you have all of the options covered?
If you are unsure, conduct a pretest using "Other (please specify)" as an option.
When building multiple choice survey questions, make sure to cover at least 90% of the respondent answers so thy data shall be clean.
7. Thou shalt use unbalanced listings skillfully
Unbalanced likert scales may be appropriate for some situations and biased in others.
Biased example: When measuring alcohol consumption patterns, one study used a quantity scale that made the heavy drinker appear in the middle of the scale with the polar ends reflecting no consumption and an impossible amount to consume. This is a sin.
Appropriate example: We expect all hotels to offer good care and may use a scale of excellent, very good, good, and fair. We do not expect poor care.
8. Thou shalt abolish double barreled questions
What is the fastest and most economical Internet service for you?
The fastest is certainly not the most economical. double barreled questions will lead to inaccurate data since your respondents may not know which part to answer.
Thou shalt never ask two questions simultaneously.
9. Honor thy dichotomous questions
When building a survey, answers should always be independent.
The question "Do you think basketball players as being independent agents or as employees of their team?" is not dichotomous. Many believe that basketball players are both.
10. Thou shalt use long questions wisely
Multiple choice questions are the longest and most complex. Free text answers are the shortest and easiest to answer. When you increase the length of questions and surveys, you decrease the chance of receiving a completed response.
Start Creating Surveys for Free
May 21, 2020
How the Utah State Board of Education is winning funding and addressing the COVID-19 crisis
May 8, 2020