Academic Experience

The dos and don’ts of ranking questions

Ranking questions are an important asset in the survey designers’ toolbox, but they can be tricky to use effectively. Researchers often want to ask respondents to rank huge lists to see what people care most about. This is a good way to get bad data.

That said, ranking questions have an important role to play so let’s explore how and when to use them.

In general, people are good at identifying a few things they prefer and a few they don’t. When ranking a list, you can expect reasonably reliable rankings for the top three and bottom three items. However, the middle rankings tend to be much more noisy and unreliable.

Try to limit your ranking question options to 6 to 10 items, and don’t read too much into the middle rankings. For example, in a list of 10 items, the difference between a ranking of 4 and 5 isn’t terribly strong or reliable for most people. Keeping this in mind when you design your rating tasks will help you not only design better questions, but also draw better insights from your data.

If you have a large list of items that you would like to have ranked, consider using broad categories to group the items. For example, if you have a list of 35 product attributes, try to think of 4 or 5 categories that contain all of those attributes and ask respondents to rank the items within each category and then rank the categories overall.

This approach of breaking large ranking tasks into smaller, more manageable tasks will result in a greater number of questions that each respondent must answer. However, the overall task will be much easier for respondents. It will also provide you with a more detailed and insightful dataset, which will help you make better decisions.

Depending on what you are asking respondents and the device-types respondents will be using, Qualtrics can help you implement ranking questions in a variety of ways. For example, drag and drop questions are likely easier for respondents who are using desktop computers than for respondents who are answering on a smartphone. Be sure to pre-test different types of ranking formats on different devices to see which is the best fit for your survey.

Free eBook: The Qualtrics Handbook of Question Design