Guide
Metatheory and the Escalating Measurement Quality Issues in Online Survey Research
Researchers have forever struggled with measuring that one single concept that provides the silver bullet for describing, understanding, or predicting that all important concept of interest. For survey researchers, concepts must be identified, understood, with relationships identified and measured. Metatheory, the investigation of investigation, is a critical part of being a superior researcher who understands not only what is being measured, but why.
Metatheory may be broadly defined as the theory of investigation. In survey research, metatheory gives understanding and interpretation to that all important process of reducing abstract concepts to observation and measurement.
Surveys are conducted with the goal of providing an operational interpretation of the concepts of interest. Surveys apply measurement, or the “assignment of numbers to respondents to represent amounts or degrees of a property (concept) possessed by all of the respondents.” This measurement provides interpretation of the properties of persons, cognitions, behaviors, emotions, events, objects, relationships, and many other “concepts” of interest to the researcher. Metatheory is a prized resource for preparing a well designed, organized, and meaningful survey.
Modeling for Completeness Metatheory involves the organization of the concepts being investigated into models that represent relationships. We are all familiar with the use of models, whether model airplane kits, drawings of buildings, descriptive text and flowcharts, or icons on a computer screen. The most fundamental issue in modeling is the convergence between the model and the reality it is designed to represent. This applies to both the process being modeled as well as the components that are part of that process. Models are intended to represent reality.
The most exceptional and powerful surveys are not hodge-podge lists of questions, but are questions built around a model of what is being investigated. Survey questions are most effective when they focus on concepts that model a process or components and confidently represent reality on all significant issues.
Two of the most useful measures of reality are called “validity” and “utility”. Validity refers to the accuracy of the survey model in describing and predicting reality. A sales forecasting model which does not forecast sales with reasonable accuracy is probably worse than no sales forecasting model at all.
Utility is a different issue. Survey development often mirrors mathematical predictive models that are not incomplete, but too complete. Model developers often try to achieve validity, but are led to include so many variables (with correspondingly difficult data collection problems), that the basic structure of the model becomes buried, input data costs become escalated, and confidence in the model is lost. The initial model may have been reasonably valid, but the resulting operationalization has lost utility because of slow decision making and increased cost. Survey research and survey building are exactly the same – more may produce less.
Survey research is often excused from not representing reality perfectly and, in fact, benefits when the research is simple enough for the managers to understand and deal with. Survey questions focusing on a specific topic or aspect of a model should be evaluated to help us arrive at results more quickly, with less expense, or with more validity. However, research used to help make hundred-million-dollar decisions should be more complete than that used to make hundred-dollar decisions. The completeness and validity required in a survey depends on the required accuracy of results.
The building blocks for survey models of reality are concepts, variables, operational definitions, and propositions. Let us take a brief look at each of these.
CONCEPTS
A concept is an abstraction formed by generalization about particulars. We can ask questions about “attitudes,” “intentions,” and “loyalty.” These are all concepts, as are “advertising effectiveness,” “consumer satisfaction,” and “consumer price elasticity.” We refer to “consumer attitude” as a concept, but the researcher must also view it as a “construct” that can be observed, measured, and related to other constructs in a model that help us understand the behavior of our customers.
VARIABLES
Survey researchers loosely call the measurable and quantifiable constructs they study “variables.” Treated as a variable, “consumer attitudes” are measured to produce data that represents the consumer attitudes concept.
OPERATIONAL DEFINITIONS
An operational definition assigns meaning to a variable by specifying how it is to be measured. It is a set of instructions about how we are going to treat a variable. We can talk about “consumer attitudes” as if we know what it means, but the term makes no sense at all until we define it in a specific, measurable way.
For example, suppose we were interested in “purchase intentions” for “Sparkle” brand window cleaner. We might operationally define the variable as the answer to the following question:
However, we could have chosen to operationally define “purchase intention” in other ways. For example, we could have used the concepts of “attitudes” and “beliefs,” which have been shown to predict purchase intention, and used a simple multi-attribute decision model:
Where:
A0 = Overall Attitudinal evaluation of “Sparkle”
Ai = Attitudes about “Sparkle” performance on the i window cleaner attributes
Bi = Beliefs about the importance window cleaner attributes to the potential customer
In this case, multiple variables are involved, perhaps measured as “acceptability of performance” and “importance to you.” Furthermore, they are related as defined by a proposition or statement of relationship between the variables.
PROPOSITIONS
A proposition is an explicit statement of the relationship between variables, including both the variables influencing the relationship and the form of the relationship. It is not enough to simply state that the concept “purchase” is a function of the concept “attitudes.” More appropriately, any intervening variables must be specified, along with the relevant ranges for the effect, including saturation and threshold effects, and the symbolic form of the relationship.
A proposition, as you can see, is quite close to a model. It is produced by linking propositions together in a way that gives us a meaningful explanation for a system or process. A model is more complete and includes multiple propositions and relationships.
Survey building involves research concepts that are identified and understood, with relationships specified and measured. The sophistication of a survey’s design will determine the sophistication of the data, and the subtlety required of their analysis. The Qualtrics survey engine, data management system, and analysis capabilities were built to enable this level of sophistication. Again, the superior researcher understands not only what is being measured, but why, and has the tools to complete the research quickly and easily.
Recent and pressing needs for enterprise-wide survey solutions have fueled an explosive growth in the industry. Over the next few years we anticipate that survey software will become an even more popular corporate solution for gathering information. Qualtrics strives to be at the forefront of online survey innovation.
Qualtrics.com provides the most advanced online survey building, data collection (via panels or corporate / personal contacts), real-time view of survey results, and advanced “dashboard reporting tools”. Qualtrics is “easy enough for an intern, sophisticated enough for a Ph.D.”
Scott Smith is the founder of Qualtrics.com. He is the James Passey Professor of Marketing and Director of the Institute of Marketing at Brigham Young University. He received his Ph.D. in Marketing and Quantitative Methods from Pennsylvania State University.