Loading...
  • Customer Experience
    Customer Experience
  • Employee Experience
    Employee Experience
  • Brand Experience
    Brand Experience
  • Product Experience
    Product Experience
  • Core XM
    Core XM
  • Design XM
    Design XM

Survey Methodology & Compliance Best Practices

What's on This Page:


Was this helpful?


This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

The feedback you submit here is used only to help improve this page.

That’s great! Thank you for your feedback!

Thank you for your feedback!


About ExpertReview

ExpertReview is a feature in the Qualtrics platform that checks the overall quality of your survey before you distribute it. It will make suggestions as to how best to improve your survey to ensure that you are collecting the highest quality data possible. Below are possible suggestions that you may receive via ExpertReview and ways that you can improve each one.

Predicted Duration

Our data indicates that surveys longer than 12 minutes (and 9 minutes on mobile) start to see substantial levels of respondent break-off. Sometimes long surveys are necessary, but in order to increase your survey completion rate, we’d suggest that you make sure this is one of those rare cases.

Qtip: The metric calculated in ExpertReview takes into account both the longest and shortest paths possible in your survey (meaning that it accounts for your Survey Flow as well). It estimates the longest path based off of the following:

  • Average human reading speed of 300 words/minute
  • 500 milliseconds time to transition between questions on a page
  • 300 milliseconds time to transition between choices in a question
  • 3 seconds time to transition between pages
  • 2 seconds time to select answer for a slider question
  • 10 seconds to type in a text-entry response
Qtip: For the MaxDiff and Conjoint questions used in ProductXM, the predicted duration may vary further due to the way these questions are typically answered by respondents.

A quick look at a survey containing page after page of boring questions produces a response of “there is no way I’m going to complete this thing.” If a questionnaire is long, the person must either be very interested in the topic, a true bleeding heart, an employee, or paid for their time. Internet surveys have some advantage because the respondent will often not see all of the survey at once. However if your survey sends them page after page of questions, your response rate will drop off dramatically.

In addition to shortening your survey’s length, we recommend keeping the following in mind when trying to increase your response rate:

  • Let respondents know up front how long to expect it will take them to complete the survey.
  • Using a progress bar can actually decrease completion rates.
    • Instead, it’s better to use textual cues (i.e., “We have just a few questions left!”).

Not sure how to go about shortening your survey? Check out our support page on Deleting Questions.

Mobile Compatibility

Up to 53% of surveys started on Qualtrics happen on mobile devices. Unfortunately, many of these mobile respondents end up leaving before they finish the survey. Making sure your survey is “mobile-friendly” can help increase both completion rates and the representativeness of your data.

Not sure how to go about making your survey mobile compatible? Check out our support page on Mobile Survey Optimization.

Number of Text Entry Boxes

Writing text takes a comparatively large amount of mental energy for respondents. Once a survey has more than three open-text boxes, we find that, on average, completion rates begin to decline and respondents start writing a lot less text in their responses. You may want to make sure that you need these many open-text response questions in your survey before fielding it.

Before using a text entry question, there are some very clear costs that you should take into consideration:

  • They take more time for respondents to answer.
  • Respondents often don’t provide very valuable textual responses in web surveys.
  • You will have to code or analyze the responses.
  • There is a possibility for drastic variance and/or bias.
  • Ultimately, this type of data collection will result in more work for you.

The reason that open-ended questions are not used more often is that respondents generally do not like them very much. However, if you use open-ended questions judiciously, it is possible to avoid both excessive cognitive demand and long completion times. Asking open-ended questions that are very specific and easy to answer will allow you to realize the benefits of this powerful question type and avoid annoying or fatiguing your respondents.

For example, in many cases, when a number is being requested, it is best to use an open-text question. Asking for a person’s age in years and letting them type the number is easier for the respondent to do than selecting from a drop-down list. It’s also more precise than selecting an age group. When using the validation options available in Qualtrics, you can also ensure that only plausible responses are given (e.g., you can set the text-box to only accept numbers between 18 and 100). In general, it is best to not use a lot of open-ended questions because they can lead to respondent fatigue faster than other types of questions and are more likely to induce your respondents to leave the survey before completing it.

Not sure how to go about removing a text entry question or “Allow Text Entry” box? Check out our support pages on Deleting Questions and Allow Text Entry.

Number of Matrix Tables

Did you know that respondents don’t like matrix tables? Research has shown that response quality and completion rates both decline when questions use the matrix format. We’ve detected that the number of matrix rows contained in your survey may negatively impact the quality of data that you collect.

We recommend simplifying possible responses (for example: Strongly Disagree, Disagree, Neutral, Agree, Strongly Agree could be simplified to Disagree, Neutral, Agree). Use separate multiple choice questions where applicable, and try to keep matrix tables to a minimum.

Number of Grammatical Conjunctions (“And”, “Or”)

Did you know that including grammatical conjunctions like “and” or “or” in a question typically means that it is double-barreled? Double-barreled questions confuse respondents and make your analysis invalid. Be sure that you are clear and specific in your directions and in your questions.

Qtip: Asking two questions in one is called a double-barreled question. For example, “What is the fastest and most economical Internet service for you?” The fastest is not always the most economical.

ExpertReview will detect questions that might be problematic because they contain a grammatical conjunction. In order to have valid results, you may want to either ask about a single subject or make two questions if you need to know about two subjects.

Not sure how to go about copying a question and then editing it? Check out our support page on Creating Questions.

Accessibility: WCAG

ExpertReview will indicate if one or more questions in your survey are not WCAG AA/508 accessible. While you may ignore this if it is not a requirement for your organization for sending out surveys to your respondents, we always recommend making your surveys as accessible as possible.

Making your survey accessible ensures that many different demographics of respondents are able to access and complete your survey. Inaccessible surveys could unintentionally exclude potential respondents from your research.

Attention: Accessibility of questions is determined by the type of question you use. ExpertReview does not evaluate if the look and feel of questions is accessible. Ultimately, you are responsible for verifying the accessibility of your own survey.

Not sure how to go about ensuring that your survey is accessible? Visit our Check Survey Accessibility support page for a list of features we offer that are WCAG compliant.

Question Readability

The harder a question is to read, the longer it takes respondents to finish your survey, and the more likely they are to abandon the survey or provide low quality answers. Questions can be deemed hard to read based off of the complexity of the vocabulary used and the length of the question.

The following can help make a question more readable:

  1. Don’t use a multi-syllable word where a shorter one will do.
    Example: “Was your stay satisfactory?” vs. “Did you enjoy your stay?”
  2. Keep questions concise and to the point. Here are some tips to help you:
    • Do not add unnecessary details to the question. It doesn’t hurt to establish context, but don’t go overboard.
      Example: Some of our respondents asked us in the past to make it easier for big groups and families traveling together to get discounted offerings. Well, we are now allowing you to book two rooms for the price of one! On a scale from one to ten, how likely are you to purchase this offering?
    • Strike out unnecessary clauses and words where you can.
      Example: “How can our staff make sure that your stay in our hotel is as enjoyable as possible?” vs. “How can we make your stay more enjoyable?”
      Qtip: Adverbs and filler words like “that” can almost always be eliminated without sacrificing a sentence’s meaning.
    • The passive voice calls for longer clauses than the active voice. Use the active voice instead.
Qtip: Question readability does not assess font colors, contrast from the background, or font type.

References

Much of the survey methodology content on this page is contributed from The Qualtrics Handbook of Question Design by David L. Vanette, Principal Research Scientist at Qualtrics. To access the entire handbook, visit this web page.