About Expert Review
Expert Review is a feature in the Qualtrics platform that checks the overall quality of your survey before you distribute it. It will make suggestions as to how best to improve your survey to ensure that you are collecting the highest quality data possible. Below are possible error messages that you may receive via Expert Review and ways that you can resolve each one.
Invalid Display Logic
If Expert Review has indicated the use of “invalid display logic,” your display logic may be referencing a question later in the survey, a deleted question, or another deleted item.
Below are some common ways people set up display logic incorrectly, and how to correct this. For a guide on creating display logic from scratch, visit the Display Logic support page.
Order of Logic
The order of your questions is very important when setting up display logic. Display logic can only hide or reveal questions based on previously answered questions. Another way to think of it is this: display logic cannot look into the future and tell what respondents will say to a question they haven’t answered yet. Even if the question is on the same page, the order is important.
Invalid Logic appears on your question when you’ve deleted the question being used in the display logic.
If you do delete an important question, you may still be able to retrieve it from your trash. Scroll to the very bottom of the survey, expand the trash, click the question, and click Restore in the upper-left. If this is the correct question, it will automatically fix the display logic.
If the question is not in the Trash, or if the Invalid Logic error remains, you will have to rewrite the display logic. Once you’ve created your new question or decided on a different question you want to base the logic off, open the dropdown and select Edit.
Qualtrics will not tell you if you’ve deleted a quota being used in logic. So first, check to see if there’s a match between what the display logic says and what’s on your quotas page. (Quotas can be found in the survey options.)
If you are still having troubles with the display logic, click the dropdown arrow on the blue banner and select Edit. The field that should display the quota’s name will be empty if the original quota has been deleted. You will have to recreate the quota and rewrite the display logic.
Deleted Embedded Data
If you delete embedded data from your survey flow, the system won’t warn you about connected display logic. Make sure that the embedded data used in your display logic has a match in the survey flow. Once this is done, you do not have to edit the display logic, so long as the spelling, punctuation, spacing, and capitalization is the same. For more on adding embedded data to the survey flow, see the linked support page.
Display Logic in Branches
While you can use branching and display logic together, you will need to be careful that branching does not unintentionally invalidate your display logic. It is possible to create display logic based on a question that is only shown in one survey path. Any respondents who do not go down this survey path will not see the subsequent questions with the display logic. Expert Review will flag these cases for you to review before publishing your survey. Additionally, if you have multiple branches with the same condition, Expert Review will flag the latter branches. This is due to the fact that the logic parameter can change in between branches (for example, if an embedded data value is set and then overwritten in between branches and the Embedded Data field is used in the logic).
Q2 asks respondents if they like to ski, but this question is only in one region’s block. Later on, Q3 has display logic set to display if respondents answer “Yes” to Q2. Since Q2 is located in just one region’s block, only respondents from that block will see Q3 even though this question was meant to be asked of all respondents.
Invalid Piped Text
If Expert Review has indicated the use of “invalid piped text,” then your piped text may be referencing a question on the same page, a question later in the survey, a deleted question, or another deleted item.
Here are some common ways people set up piped text incorrectly, and how to correct this. For a guide on adding piped text from scratch, see the linked support page.
Order of Piped Text
If you are piping one answer or value into another part of the survey, you must make sure that the following conditions are met:
- The answer being piped in and the place where it’s being piped are on separate pages of the survey.
- The answer being piped in is asked for before it’s being displayed.
What this means is that Qualtrics can’t pipe in an answer until it knows what it is; this includes submitting the answer so the survey has officially saved it and can use it on future pages. So when piping text in, you want to do the following:
- Create the question whose answer will be displayed elsewhere in the survey.
- Add a page break. This can be done with page breaks, blocks, the look and feel settings, or skip logic that happens in between the questions.
- Create the question or descriptive text where the answer will be piped.
Is your piped text not displaying a value? It’s possible you’ve deleted the item it’s trying to display data from. Here are a few ways to see if this has happened, in addition to solutions for these issues:
- Every question has an internal QID, which is used in the piped text. Copy this QID and search for it in the search bar to the upper-right. The question that appears either has this QID (or has this piped text pasted into the question/choice). If the search results are blank, you’ve deleted the question.
- Quotas get an ID, too, but this is harder to search. Go to the quotas page and see if a quota of the same name exists. If it does, generate new piped text for this quota. If the ID that starts with QO_ and is followed by a string of letters and numbers matches between the new and old piped text, the quota has not been removed.
- So long as there are values set in the contact list, you can add the embedded data back to the survey flow anytime to ensure future respondents have their embedded data displayed to them. However, if you are using branch logic to set embedded data values, you may want to make sure your branches are set up correctly, as this may be the reason your piped text isn’t displaying.
End of Survey Scoring Not Set Up
Scoring categories allow you to score your respondents on a group of related questions. For example, let’s say there are 18 questions on biology and 10 questions on chemistry in your survey. Your 18 biology questions can be added together in the Biology scoring category for a total of 18 points max, separate from the Chemistry category. In addition, you can add the same questions to different categories, accounting for overlap.
On this page, you will be able to assign scored values to scoring categories. See the linked page for more details on how to add scores to questions.
Removing Score Reports
Of course, there’s a chance you don’t want to add scoring to your survey at all, or if you do, you don’t want respondents to know their score. Follow the directions to remove the score report.
Expert Review may identify an issue with the HTML in your survey. This check will validate the HTML to ensure that it will render. Expert Review should highlight the specific line of HTML that is causing issues.
This section will advise on two things: how to access the HTML editor, and how to remove HTML so you can start from scratch, without losing your plain text.
Removing HTML From One Question
In this example, the font and any other HTML that is not visible to us will be removed, but the text, “Do you have kids?” will remain.
Removing HTML From Multiple Questions At Once
If you are still editing a survey or working from a copy of an older survey, there may be times that any translations you’ve added are incomplete. Expert Review should identify the questions missing translations.
Adding a Translation to a Question
- Adding your own translations: Click into a question or answer choice text on the right side to enter the text for the translation.
- Automatic translations with Google: Click the red circle. Qualtrics will ask if you’re sure before applying Google Translate to the text.
- Selecting the language to translate: Click the dropdown next to the question number on the right. In the example above, it’s set to Spanish (Latin America). You can add or remove languages from this dropdown using the checkboxes.
Translating Multiple Fields Using a Spreadsheet
You can also use a CSV or TSV to translate multiple fields at once. See the page on Importing a Translation for more information.
Removing Unwanted Translations
Especially when working with a copy of an older survey, you may not want to use the translations in the survey. See the page on Removing a Translation for instructions.
Timing/Metadata Questions Are Alone on the Page
Common Mistakes with Timing Questions
If you are trying to time a question or set of questions, or if you are trying to limit the time respondents may spend on the page of a survey, you need to make sure that the timing question is on the same page as the questions you are trying to time and that each group of questions you’re trying to time has its own timing question paired with it. Check the Timing Question page for information on adding page breaks.
If you have questions per page customized in your look and feel and you are using timing questions, do not ever set the default number of pages to 1. Each timing question counts as one question, and will not properly time the question or questions if it appears on a page alone.
Common Mistakes with Meta Info Questions
The meta info question will not record data until the respondent reaches it and submits the page of the survey it’s on. That means if you add a meta info question to the last page of the survey, respondents who completed only part of the survey won’t have any meta info data.
In order to ensure that data for the question is recorded, be sure to place the meta info question in a block with other questions that are displayed! We typically recommend placing it as the first question in your first block of questions. This can help you avoid any logic that may cause the question to be skipped. Since the meta info question is not displayed to the respondents, they will not know the difference.
You can move a question to the top in two ways:
- Clicking, dragging, and dropping the question to the desired location.
- Open the question editing menu and click Move question. You will then be able to choose the block where you want the question to be moved and specify that it should go to the top of the block.
More Than One Timing / Meta Info Per Page
There should never be more than one timing question per page. There should also never be more than one meta info question per page. (You can add one of each to the same page, however.)
For guidance on the many ways you can add page breaks, see the Add Page Break support page.
The Last Question in the Survey is Descriptive Text
Often times people will add a text / graphic question to the end of their survey because they think it’s the best way to thank the respondents and let them know they’re done with the survey. However, this is not the case! Because this creates one more page of the survey, the respondent who reaches this page but doesn’t click the next button technically hasn’t submitted their response. This can cause issues with your data and with your completion rates.
Adding an End of Survey Message
Instead of using text / graphic questions, Qualtrics already has a message that appears after the respondent completes the survey. Completion is indicated by clicking the very last next button.
If you would like to add your own message, you can do one of the following.
- If every user should get the same message when they finish, you can simply scroll to the bottom of the survey builder, select the End of Survey block, and add a custom message to it. See the Changing the End of Survey Message page for detailed steps.
- If you are using branch logic in your survey flow to send respondents down different paths, you may want to use end of survey elements in the survey flow to add different messages to different paths. On each branch that leads to a special message, add an end of survey element and click Customize to add a new message. Select Override Survey Options and Custom End of Survey Message to create or select a new message.
Adding a Submit Button to the Last Page of the Survey
Sometimes you may wish to have a separate message right before the end of your survey and after your survey has concluded. Other times you may just want a button that clearly indicates the respondent is about to submit their survey, without adding additional instruction.
You can add a Submit button to the end of your survey by doing the following:
- The first step to adding a submit button is to put the last page of your survey on its own block. First, select the question or question you wish to have on the last page.
- Click Move to a New Block. This will move the content to a new block at the end of the survey.
- Scroll down to your new block and click on it.
- Select Next/Previous button text.
- Under Next Button Text, type Submit, or the desired text.
- Click Save.
Large Fieldset that Creates Reporting Issues
Expert Review may warn you that your survey will generate a very large fieldset that will make it difficult to use on the reporting end. What this means is that the structure of the survey adds additional fieldsets that make your exported data files gigantic and your standard reporting visualizations unable to display meaningful data.
Sometimes, your study’s design may necessitate a complex build that expands the data set. Often, this means you will have to clean the data yourself. However, this section will go over common sources of expanded fieldsets so that you can make reductions where able.
Loop and Merge
Loop and merge is a feature that repeats blocks either based on the answer to a question, or based on the number of fields the survey creator has added.
Here are a few ways to reduce fieldsets created by loop and merge:
- Ask yourself if you need loop and merge, or if a unique block for each set of follow-up questions would reduce the number of total fields. For example, let’s say you’re looping based off a question, and only 3 of the 5 choices should lead to follow-up questions. Or, let’s say there are pretty big differences in which questions are asked for each choice. Then you would want to create different blocks for each possible answer and then use branch logic to determine when respondents go to each block.
- If you are looping based off of a number, make sure you set a Max number of loops and ensure that this number isn’t too high.
- If every respondent is looping the same number of times, make sure the total number of loops isn’t too high.
- Make sure the looped block only contains questions you absolutely need to repeat.
- Remember that you can filter data, so that if respondents are only supposed to answer questions about one choice they made in a previous question, you don’t need to loop at all. Instead, try using one block of questions with piped text. Piped text can indicate the answer provided in previous questions, which tells the respondent what they are being asked follow-up questions about. Loop and merge should only be used if respondents are selecting multiple items they must then answer the same set of questions about.
Sometimes, the sheer number of branches and blocks you’ve added can expand your fieldset. When building elaborate branches, consider:
- If the difference between choices is just a few questions, consider using display logic on each question.
- Is your embedded data set up correctly? Each field should contain unique information. See the section below for more.
Having thousands of embedded data can significantly expand your dataset. Here are some ways to cut down on embedded data:
- Make sure you aren’t using multiple terms for the same thing. Figure out if you need “Date of Birth” or “D.O.B.,” but don’t use both. The same can be said between questions and embedded data. If you already asked for gender, do you really need to set embedded data for it, too?
- If you’re setting values in the survey flow, make sure you have no spelling, capitalization, punctuation, or spacing errors. The minute you misspell an embedded data field’s name, you create a new column of data.
- It can be easy to pull in the names of embedded data from your contact list. Especially if you have tons of fields to add, this can save you time! But sometimes you’ll pull in fields you know you don’t need for your study. You can cut down on data columns (but still be able to add data from the contact list back later) by deleting fields. In the Survey Flow, just click the name of the embedded data and press Backspace on your keyboard to remove it.
Drill Down Questions
The drill down question is used to help respondents choose from a long list of options. Respondents first pick from a general dropdown list and from their answers are presented with specific follow-up lists to “drill down” to their answer. Because of the multi-tiered nature of this question type, it can result in the creation of many columns of data.
The best way to reduce the size of drill down questions in data exports is to make sure you export is to export without Split multi-value fields into columns selected. There is not an equivalent for reports.
Before you collect any data, you can also consider whether multiple questions would be a better format than one drill down, especially if there is a large difference in the size of each drill down.
Number of Choices & Carry Forward
Choices do not affect the reports as much as they affect exports. The number of choices can expand the export when you have Split multi-value fields into columns selected.
One of the ways you may be able to cut down on answer choices is if you are using carry forward. Make sure you are using this option because you need it. If you don’t actually need to carry every choice forward and are only interested in a few, you should probably add the choices manually.
Remember that before your survey has been activated (so long as your survey isn’t copied from a previously active survey), you can click Edit Multiple in the question editing pane to add or delete multiple choices at a time.