Otto von Bismarck one remarked that “Laws are like sausages, it is better not to see them being made.” He obviously wasn’t a huge proponent of process transparency.


There are researchers that might advance a similar philosophy about the results or insights of their work. In some cases, this is because they’ve massaged the data through “fishing” or “data dredging,” and are either not confident in their methods or are effectively hiding aspects of their research processes in hope that they won’t be scrutinized by inquiring minds.


It’s unfortunate that this happens. Unreliable or invalid results, passed off as sound insights, can be worse than no results. Using inaccurate insights can lead to overconfidence in ill-informed decisions simply because they are “based on data” – regardless of the fact that the data don’t actually contain the insights that they’ve been forced to represent.


The Case for Transparency


Insights from data can have immense power to drive change and success. But in order for the insights from your data to be compelling, people have to believe that the data collection and analysis processes were completed in valid and reliable ways. And there’s no better way to demonstrate the validity of your data than by being totally transparent about how you got it.


This transparency means creating or transforming your research process so that there are no “black boxes.” As a researcher, it’s a best practice to document your methods as much as possible and make all of the following available for review:


  • The complete survey questionnaire
  • Any data coding or management decisions
  • All analysis procedures or code that was tested


The goal of transparency efforts should be to prove that anyone else in your organization or field could take your data and analysis documentation and reproduce your exact results. If you can do that, your insights will be much more believable.


The Goal Is To Be Right


This level of transparency into the research process makes many researchers uncomfortable. But if the goal is to be “right” with research, then adopting research strategies that emphasize transparency and reproducibility is critical.


Regardless of whether key stakeholders or decision makers take a “bottom line” approach and simply want to know what the data say, taking steps to make your research transparent will help you to be right more often­—which will help your organization to be right more often.


If you are a decision maker, it is important to implement processes that are aimed at improving research transparency in order to ensure that the results that you are being presented with are valid and reliable. It should always be a big red flag if a researcher or research team is hesitant or unwilling to adopt these practices.


Also by this author:


Survey Speeding, Part 1: Is It Harmful or Harmless?
Survey Speeding, Part 2: Designing Surveys to Avoid Harm
Why Using Grid Questions Is Probably Hurting Your Data