Common Survey Errors

About Expert Review

Expert Review is a feature in the Qualtrics platform that checks the overall quality of your survey before you distribute it. It will make suggestions as to how best to improve your survey to ensure that you are collecting the highest quality data possible. Below are possible error messages that you may receive via Expert Review and ways that you can resolve each one.

Invalid Display Logic

If Expert Review has indicated the use of “invalid display logic,” your display logic may be referencing a question later in the survey, a deleted question, or another deleted item.

Below are some common ways people set up display logic incorrectly, and how to correct this. For a guide on creating display logic from scratch, visit the Display Logic support page.

Order of Logic

The order of your questions is very important when setting up display logic. Display logic can only hide or reveal questions based on previously answered questions. Another way to think of it is this: display logic cannot look into the future and tell what respondents will say to a question they haven’t answered yet. Even if the question is on the same page, the order is important.

Example: In Q1, the survey asks if the respondent has kids. If they say yes, then Q2, asking how many kids they have, will appear if in page display logic is enabled. If in page display logic is not enabled, then Q2 will not appear until the respondent goes to the next page of the survey.

Example: Q1 asks how many kids the respondent has. But because of the display logic on the question and the order the questions appear in, respondents will only see Q1 if in page display logic is enabled. Using in page display logic this way should be avoided since display logic should be used to display follow-up questions, not preceding questions.

Deleted Questions

Invalid Logic appears on your question when you’ve deleted the question being used in the display logic.

Example: Q1, asking if the respondent has kids, has been deleted. Q2, asking how many kids the respondent has, now has an Invalid Logic message on it.

If you try to delete a question being used in display logic, Qualtrics will warn you before letting you proceed.

If you do delete an important question, you may still be able to retrieve it from your trash. Scroll to the very bottom of the survey, expand the trash, click the question, and click Restore in the upper-left. If this is the correct question, it will automatically fix the display logic.

If the question is not in the Trash, or if the Invalid Logic error remains, you will have to rewrite the display logic. Once you’ve created your new question or decided on a different question you want to base the logic off, open the dropdown and select Edit.

Deleted Quotas

Qualtrics will not tell you if you’ve deleted a quota being used in logic. So first, check to see if there’s a match between what the display logic says and what’s on your quotas page. (Quotas can be found in the survey options.)

If you are still having troubles with the display logic, click the dropdown arrow on the blue banner and select Edit. The field that should display the quota’s name will be empty if the original quota has been deleted. You will have to recreate the quota and rewrite the display logic.

Example: The Provo Citizens quota has been deleted, so when you go to edit the quota, it says to Select a quota instead of saying Provo Citizens.

Deleted Embedded Data

If you delete embedded data from your survey flow, the system won’t warn you about connected display logic. Make sure that the embedded data used in your display logic has a match in the survey flow. Once this is done, you do not have to edit the display logic, so long as the spelling, punctuation, spacing, and capitalization is the same. For more on adding embedded data to the survey flow, see the linked support page.

Display Logic in Branches

While you can use branching and display logic together, you will need to be careful that branching does not unintentionally invalidate your display logic. It is possible to create display logic based on a question that is only shown in 1 survey path. Any respondents who do not go down this survey path will not see the subsequent questions with the display logic. Expert Review will flag these cases for you to review before publishing your survey. Additionally, if you have multiple branches with the same condition, Expert Review will flag the latter branches. This is due to the fact that the logic parameter can change in between branches (for example, if an embedded data value is set and then overwritten in between branches and the Embedded Data field is used in the logic).

Example: Q1 asks respondents their home region and then branches them accordingly.

Q2 asks respondents if they like to ski, but this question is only in 1 region’s block. Later on, Q3 has display logic set to display if respondents answer “Yes” to Q2. Since Q2 is located in just 1 region’s block, only respondents from that block will see Q3 even though this question was meant to be asked of all respondents.

Invalid Piped Text

If Expert Review has indicated the use of “invalid piped text,” then your piped text may be referencing a question on the same page, a question later in the survey, a deleted question, or another deleted item.

Here are some common ways people set up piped text incorrectly, and how to correct this. For a guide on adding piped text from scratch, see the linked support page.

Order of Piped Text

If you are piping 1 answer or value into another part of the survey, you must make sure that the following conditions are met:

1. The answer being piped in and the place where it’s being piped are on separate pages of the survey.
2. The answer being piped in is asked for before it’s being displayed.

What this means is that Qualtrics can’t pipe in an answer until it knows what it is; this includes submitting the answer so the survey has officially saved it and can use it on future pages. So when piping text in, you want to do the following:

1. Create the question whose answer will be displayed elsewhere in the survey.
2. Add a page break. This can be done with page breaks, blocks, the look and feel settings, or skip logic that happens in between the questions.
3. Create the question or descriptive text where the answer will be piped.

Example: In the image below, the piped text in Q6 will display the answer from Q5 because there’s a page break between the 2 questions.

Example: In the example below, the piped text in Q6 will display the answer from Q5 if In Page was not selected when setting up the display logic. If In Page was selected, however, then the piped text will be blank since Q5 and Q6 will display on the same page.

Deleted Items

Is your piped text not displaying a value? It’s possible you’ve deleted the item it’s trying to display data from. Here are a few ways to see if this has happened, in addition to solutions for these issues:

• Every question has an internal QID, which is used in the piped text. Copy this QID and search for it in the search bar to the upper-right. The question that appears either has this QID (or has this piped text pasted into the question/choice). If the search results are blank, you’ve deleted the question.
  
  
• Quotas get an ID, too, but this is harder to search. Go to the quotas page and see if a quota of the same name exists. If it does, generate new piped text for this quota. If the ID that starts with QO_ and is followed by a string of letters and numbers matches between the new and old piped text, the quota has not been removed.
  
• So long as there are values set in the contact list, you can add the embedded data back to the survey flow anytime to ensure future respondents have their embedded data displayed to them. However, if you are using branch logic to set embedded data values, you may want to make sure your branches are set up correctly, as this may be the reason your piped text isn’t displaying.

End of Survey Scoring Not Set Up

This message means you are trying to display the score at the end of the survey, but there’s no scoring on your survey.

Scoring categories allow you to score your respondents on a group of related questions. For example, let’s say there are 18 questions on biology and 10 questions on chemistry in your survey. Your 18 biology questions can be added together in the Biology scoring category for a total of 18 points max, separate from the Chemistry category. In addition, you can add the same questions to different categories, accounting for overlap.

You can navigate to Scoring by going to Survey options and selecting Scoring.

On this page, you will be able to assign scored values to scoring categories. See the linked page for more details on how to add scores to questions.

Removing Score Reports

Of course, there’s a chance you don’t want to add scoring to your survey at all, or if you do, you don’t want respondents to know their score. Follow the directions to remove the score report.

1. Navigate to Scoring by going to Survey options and selecting Scoring.
   
2. Go to Scoring Options.
   
3. Make sure that next to Show Scoring Summary for Category, nothing is selected.
   
4. Click Save.

Incomplete Translation

If you are still editing a survey or working from a copy of an older survey, there may be times that any translations you’ve added are incomplete. Expert Review should identify the questions missing translations.

This section will discuss how to add missing translations and how to remove unwanted translations. Navigate to translations by going to Survey options and Translations.

Adding a Translation to a Question

Use the arrows to navigate between questions. The circle containing a percentage of completion will be red when a question isn’t translated, or green when it is completely translated.

• Adding your own translations: Click into a question or answer choice text on the right side to enter the text for the translation.
• Automatic translations with Google: Click the red circle. Qualtrics will ask if you’re sure before applying Google Translate to the text.
• Selecting the language to translate: Click the dropdown next to the question number on the right. In the example above, it’s set to Spanish (Latin America). You can add or remove languages from this dropdown using the checkboxes.

Translating Multiple Fields Using a Spreadsheet

You can also use a CSV or TSV to translate multiple fields at once. See the page on Importing a Translation for more information.

Removing Unwanted Translations

Especially when working with a copy of an older survey, you may not want to use the translations in the survey. See the page on Removing a Translation for instructions.

Timing/Metadata Questions Are Alone on the Page

Sometimes, the way your timing or meta info question has been set up may negatively impact your survey.

Common Mistakes with Timing Questions

If you are trying to time a question or set of questions, or if you are trying to limit the time respondents may spend on the page of a survey, you need to make sure that the timing question is on the same page as the questions you are trying to time and that each group of questions you’re trying to time has its own timing question paired with it. Check the Timing Question page for information on adding page breaks.

If you have questions per page customized in your look and feel and you are using timing questions, do not ever set the default number of pages to 1. Each timing question counts as 1 question, and will not properly time the question or questions if it appears on a page alone.

Common Mistakes with Meta Info Questions

The meta info question will not record data until the respondent reaches it and submits the page of the survey it’s on. That means if you add a meta info question to the last page of the survey, respondents who completed only part of the survey won’t have any meta info data.

In order to ensure that data for the question is recorded, be sure to place the meta info question in a block with other questions that are displayed! We typically recommend placing it as the first question in your first block of questions. This can help you avoid any logic that may cause the question to be skipped. Since the meta info question is not displayed to the respondents, they will not know the difference.

You can move a question to the top in 2 ways:

• Clicking, dragging, and dropping the question to the desired location.
• Open the question editing menu and click Move question. You will then be able to choose the block where you want the question to be moved and specify that it should go to the top of the block.

Example: Here, we put it at the top of the Default Question Block because that’s the first block in the survey.

More Than 1 Timing / Meta Info Per Page

There should never be more than 1 timing question per page. There should also never be more than 1 meta info question per page. (You can add 1 of each to the same page, however.)

For guidance on the many ways you can add page breaks, see the Add Page Break support page.