Sources of Error in Survey Research
Our post last week outlined the steps of conducting a good survey. This week, we turn to sources of errors in survey research.
In this context, errors should not be interpreted to mean “mistakes” – rather, errors are sources of uncertainty, both in the estimates in the data and the inferences about the results. Last week, we discussed how the goal of a survey is usually to make inference to a larger population of interest. Evaluations of survey data quality typically reflect the degree of success in that effort.
Survey errors reduce, but don’t necessarily eliminate, our ability to accurately make inference to the larger population. Consequently, understanding survey errors is key to understanding survey data quality. Increasing error typically results in larger confidence intervals (reduced certainty) around the estimates in the data and inferences made about the population of interest. If these confidence intervals grow too large, the quality of the data and inferences can be degraded to the point of making them uninformative.
The Total Survey Error (TSE) model** is a helpful conceptual framework for understanding sources of error and their effects on survey estimates and inferences. In this framework, the mean square error (MSE) is used to sum all of the variable errors and biases for a particular survey. These errors are specific to a survey estimate or statistic, and in practice the MSE is rarely measured comprehensively and precisely, but the goal is to estimate the MSE as accurately as possible.
Using the TSE framework, survey errors can be classified in three broad categories illustrated in the figure below.
For example, to reduce nonresponse error a researcher may devote a larger portion of her budget to incentives, but this budgetary decision will have implications for sample size which affects other sources of error. When applying the TSE framework to survey design decisions, it is important to make every tradeoff explicitly and with as much information as possible. This will allow you to assess and account for the level of error associated with each design decision. The goal for most researchers will be to minimize error (maximize quality) within the constraints of a particular budget.
To determine the approach that will minimize TSE, the researcher must assess the likely level of error for each possible alternative procedure in the flow of survey design. In the figure below, we have added the likely sources of error to our survey data collection flow chart from last week to highlight the considerations and tradeoffs.