Qualtrics Blog

Most of our clients use Qualtrics with the goal of making inference to a certain population. Sometimes it is finding out what America thinks of President Obama and other times it is gauging the interest of a new product among general consumers. Regardless of the question we are trying to answer, getting the correct sample is vital justifying the conclusions. We need to start with the right population. If you are interested in students’ thoughts on buying textbooks online for example, the population of interest is students and not all adults. Once you have the right population you need to make sure you have enough respondents to validate your findings. Here is a simple table. ‘Population’ represents the number of people in your population of interest. ‘Sample Size’ is how many respondents you need to make good inference to that population.

Population            Sample Size

100                       80

500                      217

1,000                   285

5,000                   357

10,000                 370

50,000                 381

1,000,000           384

Keep in mind that response rates for internet surveys can range from 10% to 50% so plan accordingly as you choose your panel.

One of the big question marks in business is understanding what customers want and what they prefer.  Determining trade-offs that customers are willing to make and what the optimal features are within a product is essential to success.  Developing and producing a good or service and then having customers not respond is a giant problem. The solution: a conjoint study.  Conjoint Analysis is a statistical technique that allows you to quantitatively assess the relative importance of individual components of a product or a marketing strategy.  It magnifies the joint effects of multiple product characteristics.  It can predict the customer switch rate from one product to another and will allow you to understand what features or attributes are most important to the consumer.  It can predict the reaction to new strategies and products.  It can predict the customer response to alternative pricing strategies.  It is a great tool that aids in decision making.

Conjoint analysis starts off with a survey of the customer base you are targeting.  Qualtrics.com has conjoint functionality with a great template that really simplifies the process of putting a conjoint survey together.  It is under their "advanced elements" option.  They are a company full or bright minds that can assist you from the design to the analysis and implementation.

As a web-based tool, Qualtrics possesses the ability to be manipulated with multiple programming languages. Code can be used to alter the display of text and buttons, as well as performing various functions that could not otherwise be accessed in the software.
Adding a page timer is a clever way to influence the flow of the survey, while adding some spice to the respondent’s experience.  It will remove the next button on a page and force the respondent to advance based upon a timer. On the page you would like to time, this code can be placed in the ‘code view’ of a question text. The code is as follows:

<script>
var displayTime = 10;
Event.observe(window, 'load', function() {if($('NextButton'))$('NextButton').hide();new PeriodicalExecuter(function(){if($('NextButton')){$('NextButton').show();$('NextButton').click();}},displayTime);});
</script>

The ‘displayTime’ can be changed to any number seconds. This method of survey alteration can be used to either add variance to the regular flow of questions or ensure that respondents are spending enough time on a specified page.

While conducting survey research, most academic and private sector organizations will adhere to the code of ethics and practices established by the American Association of Public Opinion Research (AAPOR). This code calls for honesty, respect, and integrity in dealing with respondents, clients, and the public.

In this code of ethics, respondents should be given the content, sponsorship, and purpose of the survey so that they may make an informed judgment about whether they wish to participate. Any assurances, such as confidentiality or anonymity, must also be kept by the researcher.

Additionally, researchers are called to disclose fully to those who sponsor surveys the limitations and shortcomings of the survey and to avoid use of methods that deliberately introduce bias into the results. A survey report should include information on who sponsored it, who conducted it, exact wording and sequencing of questions, description of the population and how a sample was selected, sample sizes and sampling tolerance, and the method place and dates of data collection.

This information is seldom available in published research reports or media summaries, but should be obtainable with a phone call or letter to the sponsor of the survey. Today, online survey software has made this process of gathering the ethical information extremely easy. With a good survey software program, one can provide the relevant information and adhere to the code of ethics and practices with ease.

I once had a music teacher who in rage and frustration would shout "do what I MEAN, not what I SAY." And while the saying was a joke, it has proved to be a truism that many people rely on.

Humans, can guess at the meaning of a statement and make sense out of what is not clear or explicitly stated. However, what must be remembered is that computers cannot. Therefore, one must speak to them as one would to a teenager - explicitly stating every last detail, leaving nothing to be assumed.

To illustrate the problems with "do what I mean, not what I say" in terms of Quotas, let’s use the simple example of setting up an enforced gender split on a survey. You need 50 males, and 50 Females. Everyone knows how to start the process and does so correctly, and the quotas themselves are set up with proper logic - If question "what is your gender" "male" "is selected" then increment the quota. The total is set to fifty, and then the action is selected. This is the most common problem in Quota set- up

"If male is selected 50 times, then deactivate the survey"

The goal is to have 50 male responses, and 50 female responses, so it makes sense that once the male quota has been met we would like to deactivate the survey for all male respondents. However, this isn't what has been stated. The above statement says that once 50 male responses have been collected the survey should be turned off for everyone. It doesn't matter if you have 10 female responses, the link itself has been deactivated, and no one can access the survey.

So the next step is to handle the quota termination in survey flow. You enter the flow, put two branches after the demographic block, one for males, and one for females and in the branches you have end of survey elements. Here is the second most common mistake in quota logic:

"If quota male has been met" - End of Survey
"If quota female has been met" - End of Survey

You would think that the computer should be able to guess at what you mean at this point. The person has already selected that they are a male or a female and should only be evaluated on the branch which pertains to them. Well… you are wrong. The respondent will be evaluated at each block, and if the logic only refers to the quota, then once one quota is met, everyone will be terminated.

Thus the correct logic for this scenario is to have a male and a female block in the survey flow, with the logic applied:

"If question 'what is your gender' 'male' is selected AND quota 'male' has been met"
"If question 'what is your gender' 'female' is selected AND quota 'female' has been met"

The conjunct statement first evaluates their gender and directs them to one branch, which then evaluates if the corresponding quota has been met. This will allow the gender whose quota is not met to continue with the survey, while the other gender has been met.

Qualtrics isn't going away, and in fact, it is continually being wound by the sharpest minds in web and computer programming. With respect to improvement, the Development Team deploys updates on a monthly basis, and since Qualtrics is a web-hosted software, there's no need for downloading or installing updates on a personal or work computer. The ease of updating allows the user to experience smooth updates day after day, month after month, and year after year.

Imagine this…The user logs into Qualtrics to check its capability on a certain feature; soon after finding what he or she is looking for, some expression like this follows: "Oh, wow; that's a lot easier than I thought." Expressions like these are not merely imaginary, they are real. Users find solutions to their most complex needs everyday, and this mostly due to updates. Updates are based primarily upon the needs of users, but the Development Team's cleverness and skill compels them to create for future needs - ahead of the curve.

As you may already know, Qualtrics wasn't meant to become the company that it is today - it happened serendipitously. It began with a Ph.D needing a tool for his market research; bright minds were gathered to create a tool based upon the needs of a Ph.D researcher, and from that small start, it has become the premiere survey software, used by the world's leading universities and corporations.

Qualtrics hasn't slowed down since it's beginning, and it will continuously improve through implementing monthly updates that are based upon demanding research needs.