Qualtrics Blog

Most researchers struggle while attempting to create a great survey. An effective survey begins with an understanding of the research process, and the first, most important step is proper formulating of the problem/opportunity to be examined. If the problem analysis is setup incorrectly, the survey questions will follow, resulting in data that is not optimal — or even worse, not actionable.The best approach to build a survey is outlined in the following steps:

1. Review the basic research objectives of the study:
   • What core problem/opportunity will this survey data help to resolve?
   • What actions will take place as a result of the completed survey data?
2. Visualize the desired information outcomes
   • What will the output reports look like?
   • What charts and graphs will be prepared?
   • What information is needed to assure that action is warranted?
3. Prepare a written list of the topics in steps 1 and 2 and order them according to their value in solving the research problem.
   • List the most important topics first.
   • ; Revisit items 1 and 2 to make sure the desired objectives, topics and information are appropriate.
4. Try to think of the respondent's point of view.
   • How easy or difficult is it for the respondent to provide information on each topic? If it is difficult, i.e. the respondent doesn't know, can't remember or won't take the time to answer, try brainstorming other ways to obtain the information — perhaps through a different question or another data collection technique.
5. Review the sequence of topics to make sure they are unbiased.
   • Do the questions asked influence or bias the results of the following questions? Oftentimes, providing too much information, or disclosing the purpose of the study can create bias.

At this point, the information needed to build an effective survey has been collected. The focus now shifts to asking valid questions and building the survey instrument.

6. Determine the type of question that is best suited for each topic, considering the type of data produced by each question type — do the question and answer formats provide enough robust information to meet your analysis requirements? Take into account the following criteria when determining what question type to use:
   • Question type:
   • Open-ended text questions, dichotomous, multiple choice, rank order, multiple choice matrix, Likert or Semantic Differential scales, constant sum, conjoint, or side by side
   • Answer Format:
   • Categorical, ordered, ranking, and rating
   • Type of Analysis to be Conducted:
   • Percentages, means and standard deviations, cross tabulations, and statistical tests
7. Draft the survey questions. Several versions of each question may be needed when building the survey. Creating both great questions and a great survey can often take five or more drafts — be patient and select the questions that provide the most actionable data.
8. 8. Review the question sequence again for bias and logical flow.
9. 9. Repeat all of the steps above to find any major holes.
   • Will the answers provide you with data that helps/solves your core problem/opportunity?
   • Ensure that more that one set of educated eyes reviews the survey before it is distributed.
   • Will each question make sense to the respondent? (Will they understand what it being asked?)
10. Time the length of the survey. A survey should take less than five to ten for a respondent to complete. At about six questions per minute, and depending on question difficulty, a typical survey is limited to about 30-40 questions. When building a survey, remember that one open-ended text question counts for three multiple choice questions.
11. Pretest the complete survey draft to about 20 or more people. Obtain detailed feedback — critically looking at their responses to ensure the respondents understand what the questions are asking and are responding accordingly.
    • Did they have questions?
    • Did they have trouble understanding what information was wanted from the question?
    • Did they take a point of view not covered in the given answer choices?
12. Revise your questionnaire and pre-test again or begin data collection.

Google's Trends tool is an interesting way to gain a bird-eye perspective on the upcoming 2008 Presidential candidates. "Trends" shows an approximation of search volume for any given search term as compared to another. In the case of presidential candidates, you can see how much each candidate is being searched for on Google in comparison with other candidates.

For example, let's look at a few of the Republican candidates, in no particular order. Only four candidates are shown here for simplicity sake, and Hillary Clinton is included for reference.

The above graph shows search volume for each presidential candidate's name in the first row, and how many times those names are being referenced in the news (or, what Google considers to be "the news") in the second. Although this data should not be considered concrete real world data, you can still draw some interesting conclusions.

One of the more interesting candidates to watch has been Ron Paul, who has surged in search volume with almost no accompanying news coverage. This is probably due to the fact that Ron Paul does very well with social websites such as Digg, Slashdot, Reddit, etc. but not as well with the general media. It is interesting to note that each Republican candidate started out with mediocre search volume at the beginning of the year, but candidates like Ron Paul and Fred Thompson have surged recently.

Here is a chart of the Democratic candidates, again in no particular order. Ron Paul is included here for comparison purposes so you can easily reference between the two charts.

Opposite to the Republican graph, leading Democratic candidates like Hillary Clinton and Barack Obama peaked at the beginning of the year and have since fallen.

The data can be further broken down by region for even more interesting information.

We can see here that people in Washington D.C. search more for Hillary Clinton, while people in Illinois search for Barack Obama more.

Further comparisons can be made by pitting the two political parties directly against eachother in Google Trends. Here we compare Clinton, Obama, Paul, and Thompson - the four candidate who have shown the most movement in search volume throughout the year.

Here is a breakdown of the above graph by city:

Here we see that the popular search goes to Paul in Austin, TX and to Obama in Chicago, IL. Compare this to the election results from 2004:

Perhaps it should already be obvious, but both data sources show Texas being heavily Republican while Illinois is heavily Democrat. This just goes to show the extent to which search volume trends correlate to real world election results. Again, none of the information should be taken as concrete because there are several reasons why search data could skew one way or the other, but the data is interesting nonetheless.

For comparison, here is a record of searches performed before the 2004 election for George Bush and John Kerry.

The overall closeness of the search volume for each candidate seemed to predict accurately the closeness of the final vote. Unfortunately data is not available for the 2000 election. It would be interesting to see how far apart search volume is in a race that wasn't quite as close.

There are many conclusions you can draw from this data, but the most important is that the election in 2008 could still go to anybody. It will certainly be interesting to watch how search volume correlates to real world election results.

Resources:
Google Trends Republican Candidates
Google Trends Democrat Candidates
Google Trends Republicans vs. Democrats

Technorati Tags: barack obama, charts, election, fred thompson, hillary clinton, joe biden, john edwards, john mccain, mitt romney, polls, ron paul, stastics