3 Tips to Get Treated, Not Tricked, by Your Survey Data
Halloween is just around the corner, and we want to keep your research cauldron brewing with the best advice to keep you from getting tricked by your data. Indeed, getting new survey data is exciting and daunting at the same time for many researchers. You know what research question you were trying to address in your survey, but what’s the best way to translate your raw data into actionable “results?”
Chances are, if you ask five researchers how they go about analyzing their survey data, you’ll get five different responses. There’s no one-size-fits-all approach, but we’ve compiled three basic rules of thumb that will help you get the most out of your data and avoid publishing misleading results.
- Visualize your data. “Seeing” your results is easier than ever with tools like Tableau, R, Excel, and, of course, Qualtrics, which enables you to visualize your data directly in the research platform. Whatever your preferred method, taking a look at your data can tell you a lot about what the results are. In many cases, visualizing your data might be all you need to do to clearly see the results of your survey.
- Don’t get hung up on statistical significance testing. Rigorous social science research is increasingly moving away from p-values and is instead focusing on whether results are “significant” or not by reporting effect sizes (how big is the real-world difference?) and confidence intervals (how certain are we about the estimates that have been generated?). In datasets from very large samples, it is common for very small real-world differences to be statistically significant because the confidence intervals around the estimates are so small. Similarly, in datasets with very small samples it is common for very large real-world differences to not be statistically significant because the confidence intervals (or standard errors) around the estimates are so big. For these reasons, focusing solely on the p-value to determine if a difference is “real” can be misleading.
- Focus on the research questions and hypotheses that drove you to collect data. Once you have a dataset it’s tempting to dredge through it to find out what “worked” in terms of generating “significant” results. With a dataset based on just 20 survey questions it is possible to make 20 choose 2 = 190 pairwise comparisons. Based on this number of comparisons and a standard alpha value of 0.05, we would expect about 10 of the comparisons to be statistically significantly different simply by chance! Focusing on your pre-specified hypotheses and research questions can help you reduce your chances of drawing conclusions or making decisions based on incorrect results.