Qualtrics Blog

Just when you think you got the hang of surveys, you sometimes find a hang-up. To avoid problems with surveys, check out three simple suggestions to help you write great questions.

Non-exhaustive listings

When you are writing multiple choice questions, sometimes it’s difficult to think of every option.

For example, if you are asking “What kind of car do you drive?” and can only think of Honda, Toyota and Ford, you might need some help thinking of other answer choices.

It is good to ask yourself: Do I have all of the options covered? If you are unsure, conduct a pretest using the "Other (please specify) __________" option.

As you read through what test respondents wrote under “other” you can go back and edit your survey before making it live to your target respondents. For example, you might have found that a lot of people wrote BMW and Jeep, you could consider putting those as options in your multiple choice question. Revising the question can help you make sure that you cover at least 90% of the respondent answers.

Unbalanced listings

Unbalanced scales may be appropriate for some situations and biased in others. When measuring alcohol consumption patterns, one study used a quantity scale that made the heavy drinker appear in the middle of the scale with the polar ends reflecting no consumption and an impossible amount to consume. A better option would be to keep all of the scale points fairly even. So instead, ask if the level of alcohol consumption is no consumption, moderate consumption, or heavy consumption.

This suggestion is displayed below:


Double barreled questions

Two questions should not be asked in the same question. For example, "What is the fastest and most economic Internet service for you?" The fastest is certainly not the most economical. Be careful to read your questions to ensure you are only asking one question at a time.

Qualtrics enables you to:

• Embed data
• Create samples and compile the results of several surveys
• Associate respondents with their responses

Although panels may initially seem foreign to users, it is helpful to view a panel as an address book.  Panels are created in a spreadsheet and are then imported into Qualtrics.  They need to adhere to a specific format, which are shown when you go select “Import/Update Panel Members” from the “Panel Members” screen.  You can add additional columns other than a name and an email address which are relevant to your study, such as age, location or gender.

Embed Data: Within Qualtrics, we refer to these additional columns as embedded data.  Embedded data can be piped into email messages, survey questions and your results.  For more information about setting up a panel, I encourage you to check out our Help and Tutorials.  Our support team is also available by phone or email.

Create Samples: One helpful panel function is the ability to create samples.  Qualtrics offers the ability to randomly generate a group of individuals from a panel.  You can set up a condition which determines what type of individuals should be part of the panel.  For instance, perhaps you want to send a survey out to only those students who were part of a university’s humanities department.  If student information is stored in the panel as embedded data, you can set up a panel sample of humanities students.  You can also generate a sample based on those who haven’t taken a survey before, after, or on a certain date.  These options are available in the “Panel” tab and then the “Sample Management” icon.

Once you have created your sample you can distribute a survey to the sample in the “Distribute Survey” tab then the “Send Survey” icon.  Under the “Who to send to” section, select “Panel Sample” and then locate your sample within the appropriate library.

Manage Respondents and Responses:  Perhaps you want to create panel members who have already completed a survey.  Or you may want to update existing panel members with data they provided in a survey.  We offer the ability to import or update panel members from a survey.  Within the “Panel” tab, select the panel you are working with.  Notice on the subsequent page the blue text in the upper right corner: “Import/Update Panel Members.”  If you opt to “Import/Update from a survey” you are then directed to dictate which survey and answers to questions within that survey you’d like to capture.

Adding images to your graphics library is quite simple.  We get to the Graphics Library by clicking the Library tab at the top of any page in Qualtrics.  We are taken into the Library Overview where you can see a summary of all your different libraries.  Now click the Graphics Library button (the button with mountains in it) and you will be able to see all of the images or graphics currently in your library.

To add more images, click the “Upload a New Graphic” button.  Now you’re going to select from where you will upload the graphic.  This is either from a file on your computer or from the internet.

      To upload a file, click “Choose File,” and find the location of the image on your computer, name the image, select the dimensions (default dimensions are automatically inputted), and click “Open."

      To upload from the internet, click “Use a Graphic from Web,” copy and paste the location (URL) of the image (you will see a preview of the image to be uploaded and be able to name it and assign it to a folder), and click “Save.”

Now you should see the image in your account and be able to select that image for use in any or your surveys.

Qualtrics users are constantly asking us how to write better survey questions.  Here are a few tips to remember while writing questions:

Write to Respondents:  Remember your audience when writing a survey.  Using words that are unfamiliar to them can cause problems.  Industry-specific jargon, technical data, and acronyms are not effective for all audiences.  If your audience does not understand the survey, then your results will be meaningless.  If you aren’t sure if a question is clear, ask someone from your desired population read over it and give you feedback.

Avoid Forcing Answers:  Sometimes respondents may not want or be able to provide the information requested.  Privacy is an important issue to most people. Questions about income, occupation, finances, family life, personal hygiene, and beliefs (personal, political, religious) can be too intrusive and rejected by the respondent. 

It is usually a good idea to allow respondents to not answer questions that may be considered sensitive. 

Before sending your survey out to hundreds of respondents, you may want to test it with the Quick Send.

When creating a panel for a large sample, distributing your survey through the Survey Mailer saves time and energy. In contrast, the Quick Send is designed to make smaller mailings more convenient.

With Quick Send, when the mailing is sent, a panel is created based off of the email addresses and other information included for each respondent. This panel can be reviewed in the Panels tab, and will include all regular panel features such as reminders and thank you messages.

The Quick Send also provides a great way to test surveys with embedded data.  Even if you will eventually use the survey mailer to distribute your survey, test it out with the Quick Send by sending yourself a copy with the embedded data fields added.  This will allow you to review it and make changes accordingly.

One of the great additions this year to the Qualtrics Research Suite is the “Test Survey” option in the Advanced Options menu.  This allows you to ask the system to take the survey for you multiple times. This will help sort out any issues you might have with your survey before it is activated.

You can then easily remove these results by navigating to the View Results tab, clicking on “Edit Results,” and selecting it from the Advanced Options menu above your summary table.

When the test is complete, you can examine the results to make sure that they give you the type of output you are looking for.  This includes ensuring complex survey logic is performing in the way you want.  This is also helpful when experimenting with different question types in your survey.  When using a question type for the first time, it is a good idea to run a test to understand how the results will look when it is finished.

It is important to note that the survey tester will not identify errors for you.  You need to check the results and make sure that things are functioning properly.  For example, if a certain question has significantly more or less responses than you expected, then perhaps there is a logic that you can go and correct.  Or maybe you will see in the results that there was an error in coding the values.  While it may not be able to find all possible errors for you, the survey tester can be a valuable tool as you test and tweak your survey before launching it to respondents.

We are fortunate that Qualtrics users are comfortable asking us questions. Many of our questions are about writing better survey questions.  Here are a few tips to remember while writing questions:

Use Mutually Exclusive Responses:  Be sure to make your response categories mutually exclusive.  Consider the question:

What is your age?

• 18-31
• 31-40
• 40-55
• 55+

Respondents that are 31 years old have two options, as do respondents that are 40 and 55.  As a result, it is impossible to predict which category they will choose.  This can distort results and frustrate respondents.  It can be easily avoided by making responses mutually exclusive.  The following question is much better:

What is your age?

• 18-30
• 31-39
• 40-54
• 55+

This question is clear and will give us better results.

Ask Specific Questions:  Nonspecific questions can confuse respondents and influence results. 

Do you like orange juice?

• Extremely Like
• Like
• Neither Like nor Dislike
• Dislike
• Extremely Dislike

This question is very unclear.  Is it asking about taste, texture, price, or the nutritional content?  Different respondents will read this question differently.  A specific question will get more specific answers that are actionable.

How much do you like the current price of orange juice?

• Extremely Like
• Like
• Neither Like nor Dislike
• Dislike
• Extremely Dislike

This question is more specific and will get better results.  You can also include multiple questions to get more information.  (Do you like the taste of orange juice?  Do you like the nutritional content of orange juice? etc.)

Following these tips will improve your research. Check back next Wednesday for our weekly tips on writing surveys.

Many clients appreciate the rich text editor feature in Qualtrics. They can edit the text (bold, add color, etc.) to any question throughout their survey. Because of this feature, clients have asked if they can edit answer choices (underline, highlight, etc.) without a rich text editor designed specifically for the answer choices.

Although the answer choices do not have a rich text editor feature, there is a straightforward way to edit the answer choices. You can insert HTML code into your answer choice to edit them. If you are not familiar with HTML code, Qualtrics provides an easy way to find it.

Click on the question and then the rich text editor tab. Now, type in what you want for your answer choice and apply the necessary editing by making it bold, italicized, etc. Then, click on "source" within the rich text editor tab. This will allow you see what it looks like in HTML code. Copy and paste the HTML code (only the code that corresponds to your answer choice) to your answer choice. You can paste this into your answer choice, simply by clicking on your answer choice and pasting the code. Be sure to delete the code from your question box.

In summary, you can use the rich text editor that is usually used for the question, to figure out the code you want for your answer choice. Keep in mind you can do this repeatedly for each answer choice. Not only will you have the text features you want in your answer choices, but you have mastered a basic process of HTML code.

With Qualtrics you can use the Data Highlighter to automatically highlight cells based on criteria that you set.

This means you can highlight parts of your data that you want to stick out. This makes it easier to scan your data and quickly see results.

For example, if I want to highlight cells that have less than 20 responses, I would select this criteria and the Data Highlighter can be set to do this automatically.

Here at Qualtrics, we love to give you options. This means you can specify any colors imaginable and you will have a legend to keep criteria organized. Criteria can be based on frequency, row percentage, column percentage, expected frequency and/or actual-expected frequency. If you have ever used conditional formatting in Microsoft Excel, you will find that the Data Highlighter provides similar functionality.

To create custom highlighting, go to the View Results Tab, create a cross tabulation and select Data Highlighter under the Data Options menu. Here you can set one or more highlighting conditions. Again, you can highlight all cells with a frequency less than 10 in light blue and all cells with an expected frequency less than or equal to 5 in red.

You can also use one of the Data Highlighter presets options to highlight full scales or top/bottom boxes. These presets are also fully customizable.

The data highlighter makes it easy for you to analyze and draw attention to meaningful elements of your cross-tab. It’s just another way that Qualtrics makes your research that much easier.

Once you join the world of research, we know you will want to produce the best survey possible. To answer to some frequent questions, we are going to post question tips every Wednesday.

Avoid loaded or leading questions:

Slight wording changes can make a huge difference in results. For instance, the words could, should, or might can influence results by up to 20%.For example:

“The Supreme Court (could / should / might) have forced the breakup of Microsoft Corporation.”

Clearly one word difference can drastically change the meaning and interpretation of the sentence.

Strong words that represent control or action, such as prohibit, produce significant differences in results as well. For example:

“Do you believe that congress should (prohibit / restrict) insurance companies from raising rates?”

There are similar words (like restrict) that are less provocative and will generate better results.

Leading questions also cause serious problems for researchers by influencing their results. For example:

“You wouldn't want to go to Rudolpho's Restaurant for the company's annual party, would you?”

Avoiding loaded or leading questions can greatly benefit your research.

Use the Funnel Approach:

Placing questions in a logical order can help respondents. Consider using the funnel approach:

• Begin with broad general questions
• Then ask more specific questions
• Finish with simple questions (demographics) that allow the respondent to wind down

Using these simple steps will improve your research. Be sure to check back next Wednesday for more tips.