XM Knowledge Base
Brand Experience
Why Using Grid Questions Is Probably Hurting Your Data
Even for experienced researchers, intuition about survey design often runs counter to best practices outlined in survey methodology literature. One of the question types that we see used in web surveys most frequently is a particularly problematic type known as the grid or matrix question type.
In some cases, researchers use grid questions because they can’t be bothered to come up with response scales that map directly onto the construct that they want to measure for each of their questions and instead use a single response scale for many different items. But in other cases, it seems that there are some misperceptions about the effectiveness of grid questions and a general disregard or unawareness of the negative effects that using this question type can have on data quality.
We find that researchers commonly have four misperceptions about grid questions:
Misperception 1: They take less effort to answer
The first misperception is that grid questions are easier for respondents. Research literature actually indicates that they are one of the hardest question types (see Dillman et al 2009, pg 179). All else equal, we want to make it as easy as possible for our respondents to provide their best data. Using question types that are unnecessarily difficult for respondents is likely to result in early fatigue, decreased motivation, and lower data quality due to satisficing.
Misperception 2: They make questionnaires shorter
The second misperception is that grid questions make questionnaires shorter because multiple individual questions can be rolled into a single question. This is a false sense of economy—researchers should consider the length of a questionnaire in terms of how many decisions or responses a respondent is asked to make rather than the number of questions that are indicated in the survey platform.
Misperception 3: They yield quality data
The third misperception is that it’s easier for respondents to complete surveys that use the same response scale for as many of the questions as possible (most commonly the problematic agree-disagree scale). The truth is that these are considerably more difficult for respondents to cognitively process and offer thoughtful, valid and reliable responses to, even if they are able to provide their responses faster. By using grid questions, researchers are favoring fast responses over quality responses.
Misperception 4: They can’t hurt response rates
Respondents are increasingly arriving at surveys via mobile devices (up to 40% in some cases), and, due to the way they often look when compressed on small screens, grid questions are some of the worst performing questions on mobile devices. The poor presentation leads to lower response rates as respondents get frustrated and break off.
If you want more information about the evidence pertaining to grid or matrix question types in web surveys, check out these articles:
The Design of Grids in Web Surveys
Speeding in Web Surveys
Grappling with Grids
Grids vs. Individual Items