Rating or Ranking: Choosing the Best Question Type for Your Data
When it comes to surveys, how you ask each question may be just as important to your data quality as what you ask.
It can be tricky to decide what type of question to use to get the most useful data. In this post, we’ll discuss the benefits of rating scales and ranking questions and when it is best to use each.
In surveys, the most commonly used question types are rating scale questions. This is where respondents are asked to indicate their personal levels on things such as agreement, satisfaction or frequency.
Rating scale questions are best used when you want to measure your respondents’ attitude toward something. Questions that include “how much…” or “how likely…” are best when differentiation between desirable things is not necessary.
That said, rating scales are often not helpful in eliciting the fine-grained data that researchers need to make decisions. For example, if you asked how much your respondents like particular dessert items, you might find that many of them engage in something called satisficing – which basically means that they shortcut the optimal response process by choosing any acceptable answer.
Maybe these responses help you, but if you happen to be wondering which dessert your restaurant should list on the new menu, then you’re in a pickle.
This sort of non-differentiation can produce data that are not very useful—especially if you’re trying to generate a ranked set of preferences. Think of the dessert example. If you’re asking your respondents about things that may all be potentially desirable, a rating question may not work.
If you are trying to get your respondents to choose between a list of desirable things, there is an easy solution: ranking questions. This question type allows respondents to identify which objects are most and least preferred.
This is particularly helpful when you are forcing your respondents to choose between two things. While some people may like everything on the dessert menu, most will settle for whichever single dessert they prefer most.
“When life forces choices, use ranking.”
This is good life advice and especially good advice for survey building. If your respondents will face real-world choices among sets of items, it’s best to allow them to rank their choices in your survey. You can always follow up the ranking question with a rating question to evaluate the strength of the preference.