Market Research

Survey trolls, the Privacy Paradox, and Pizzagate

Sunshine Hillygus PhD, is a Professor of Political Science and Public Policy at Duke University. As part of our Market Research Visionaries series she talks us through the importance of valid responses to surveys and how people lie to researchers (and themselves).

“Over the course of my career, I've just done more surveys that I can possibly count,” says Sunshine, who started her career working in research and surveys while studying for her PhD at Stanford University.

“After Stanford, my first faculty member job was at Harvard University,” says Sunshine. “There I had the good fortune of being mentored by Gary King when he was creating the Institute for Quantitative Social Science (IQSS).

“He asked for feedback from the faculty on his plan for various initiatives and programs within IQSS. I off-handedly commented that he had covered all the various methodologies and approaches, except surveys. Even though I was an assistant professor at the time he said, ‘I'm now anointing you Director of the Program on Survey Research’.

From small acorns...

What could have started out as just a small idea grew into a larger program that was dedicated to bringing together survey researchers from across the university. Sunshine and her team provided support survey researchers so that staff and students who were doing surveys, were able to make use of kind of best practices within the field of survey methodology.

Active mission

“Then when I left Harvard and came to Duke, I created the Duke Initiative on Survey Methodology (DISM) with that same model in mind. “We want Duke surveys to benefit from knowledge in the field of survey methodology and we want to contribute knowledge to the field.

“This requires engagement with academics, government practitioners and private companies, like Qualtrics.

“I really appreciate the strong connection between Qualtrics and the academic community. I've been working with Qualtrics from the very, very early on. Easily more than 10 years. I really appreciate the responsiveness that they have had to the academic community.”

How do people really vote?

Alongside her academic survey research, Sunshine is also a political scientist who studies public opinion and political behavior. “The substantive focus of my surveys is primarily focused on understanding decision making related to voting,  so if people vote and how they vote.”

This research includes a forthcoming book, Making Young Voters: Converting Civit Attitudes into Civic Action. Did she find out anything surprising during her career as a researcher?

“It surprised me that more academics don’t take a closer look at where their data has come from.

“I often joke that scholars pretend that their data are being delivered by a data storke,” says Sunshine. “They can just download it from the website and analyze it, look for the patterns that they're interested in and go on their happy way.

“I think that what most people both in the industry and in the academic field of survey research are more cognizant of the importance of questioning where our data come from, how it was produced and if there is anything about it that might jeopardize our knowledge claims.

Trolling your survey

“A lot of my research on survey methodology has focused on issues of data quality,” says Sunshine.

Sunshine says that while there's a recognition within the industry that there are problems with people not really paying attention when they're taking surveys, there are also a much deeper problem: sometimes there are people who actively work to corrupt your data.

“There are respondents who troll your survey by giving provocative or humorous responses as a way to be funny.

“And it's something that I've found has especially messed up estimates of political misperception. It is very common for the newspaper headlines to overstate the ignorance of the public because they have overlooked survey trolling.”


Sunshine uses the example of Pizzagate. “People read a headline saying a large percentage of Americans believe in Pizzagate. But it turns out that nearly half of the people who claim that Pizzagate is "definitely true" were found to be trolling the survey.”

What can you do about trolling?

There are techniques you can use to flag survey trolls within data, Sunshine says.

“Those data-quality issues are the responsibility of the researcher. We must design surveys that people want to answer and are able to answer in a clear way.

“But we also have to actively look for respondents who are paying attention and not trying to corrupt your data,” says Sunshine. “And I do think that there is increasing recognition of the importance of looking at data quality issues in terms of thinking about the future of survey research.”

Sunshine says there are some exciting innovations on the horizon in terms of survey design and analysis with respect to data quality. “For instance being able to flag problematic respondents. I also think that more broadly there are some encouraging developments with respect to combining surveys with big data to get a clearer idea of whether people are genuine or not.”

The right tools

“I've been really excited to hear about some of the tools that Qualtrics is developing around issues of data quality that allow researchers the ability to get higher quality data,” says Sunshine. “Those tools target a variety of different sources.

“So both the tools for novice survey designers who need some assistance with question best practices – as well as the kind for the more advanced users. These help to identify low-quality respondents. I really like that.”

The Privacy Paradox

Another thing that Sunshine found surprising was that there’s not always a link between attitudes and behaviors. “That's a very important thing to keep in mind when trying to make sense of survey data,” she explains.

Sunshine says that a classic example of this is called the privacy paradox. In a survey, people will talk about how concerned they are about their privacy online, such as the information that companies or politicians have about them.

“However, these privacy concerns have almost no impact on any behaviors that we can observe.”

A similar pattern is found in politics. “There are a great many people who say they intend to vote, they're very interested in politics, they care about the election results and ultimately they don't follow through on those attitudes,” she says.

So ultimately their behavior doesn't necessarily match their attitudes? “Exactly. And that disconnect is critical to understanding politics. In my forthcoming book, we examine how this helps explain low levels of youth turnout and consider policy solutions for increasing the link between attitudes and behavior.”

Improve Survey Data Quality with Qualtrics ExpertReview