Qualtrics Blog

One of the great additions this year to the Qualtrics Research Suite is the “Test Survey” option in the Advanced Options menu.  This allows you to ask the system to take the survey for you multiple times. This will help sort out any issues you might have with your survey before it is activated.

You can then easily remove these results by navigating to the View Results tab, clicking on “Edit Results,” and selecting it from the Advanced Options menu above your summary table.

When the test is complete, you can examine the results to make sure that they give you the type of output you are looking for.  This includes ensuring complex survey logic is performing in the way you want.  This is also helpful when experimenting with different question types in your survey.  When using a question type for the first time, it is a good idea to run a test to understand how the results will look when it is finished.

It is important to note that the survey tester will not identify errors for you.  You need to check the results and make sure that things are functioning properly.  For example, if a certain question has significantly more or less responses than you expected, then perhaps there is a logic that you can go and correct.  Or maybe you will see in the results that there was an error in coding the values.  While it may not be able to find all possible errors for you, the survey tester can be a valuable tool as you test and tweak your survey before launching it to respondents.

With Qualtrics you can use the Data Highlighter to automatically highlight cells based on criteria that you set.

This means you can highlight parts of your data that you want to stick out. This makes it easier to scan your data and quickly see results.

For example, if I want to highlight cells that have less than 20 responses, I would select this criteria and the Data Highlighter can be set to do this automatically.

Here at Qualtrics, we love to give you options. This means you can specify any colors imaginable and you will have a legend to keep criteria organized. Criteria can be based on frequency, row percentage, column percentage, expected frequency and/or actual-expected frequency. If you have ever used conditional formatting in Microsoft Excel, you will find that the Data Highlighter provides similar functionality.

To create custom highlighting, go to the View Results Tab, create a cross tabulation and select Data Highlighter under the Data Options menu. Here you can set one or more highlighting conditions. Again, you can highlight all cells with a frequency less than 10 in light blue and all cells with an expected frequency less than or equal to 5 in red.

You can also use one of the Data Highlighter presets options to highlight full scales or top/bottom boxes. These presets are also fully customizable.

The data highlighter makes it easy for you to analyze and draw attention to meaningful elements of your cross-tab. It’s just another way that Qualtrics makes your research that much easier.

In-depth analysis can get really ugly if you aren’t careful - especially when the values you entered don’t look like the ones in the data file. Qualtrics has an easy way to solve this problem.

Some of the most commonly used question types are Multiple Choice and Matrix Table. Each option gets its own value so multiple options lead to multiple values. These values are assigned when an option is created; however, survey creators don’t always keep answer options in the order that they were originally entered. Perhaps you want to build a survey concerning different positions in a company. You type in: (with the values analytical software will use in parenthesis)

(1) Janitor
(2) CEO
(3) HR specialist
(4) Sales representative

Later, you realize that you want to reorder and analyze these positions according to earnings. Clicking the blue arrow on the right side of the option allows you to move these up and down until you have reordered them the way you like. Remember, the order in which you first put them is the order they were given values for analysis. Here is what it would look like if you reorder it according to earnings: (with the analytical values still in parenthesis)

(2) CEO
(4) Sales representative
(3) HR specialist
(1) Janitor

To prevent these values from carrying over to the data files, the values can be recoded.

Recoding values is simple. Just click on the purple gear box on the left side of the question and scroll over the options until you find “recode values.” A box will pop up and enable you to change the values. For example:

(1) CEO
(2) Sales representative
(3) HR specialist
(4) Janitor

This will prevent frustration when viewing the data file after the data has been collected. As always, if you have any questions about this process or any other aspect of Qualtrics, feel welcome to contact our support team who will be happy to help you.

Anonymous contains two Latin roots: a/an meaning without, and nym meaning name. Anonymous therefore means without a name. Merriam-Webster defines the word this way:  

            "Not named or identified; lacking individuality, distinction, or recognizability."

Often we receive requests that go something like this: "I want to have an anonymous link, but if someone answers negatively, I want to know who they are so that I can contact them." Obviously, the ability to contact someone due to their responses violates the definition of anonymous.

It might then be helpful to go over what anonymous links can and can't do.

Anonymous links can be sent through the mailer - this is located under advanced options >> link type

Anonymous links do record an IP address

Anonymous links cannot track who has answered - the best way to ensure anonymity is to not record any personal information. Therefore, even if a link is distributed to a panel using the mailer, Qualtrics does not record who has and who hasn't completed, started, or clicked on the survey. As such, Reminders and Thank you messages are impossible to send as the system has no idea who to send them to.

Anonymous links cannot pull embedded data into the survey. Once again, Qualtrics isn't recording who is answering the survey, and cannot therefore attach embedded data associated with a respondent to the response. John Does do not have addresses, regions, sales reps, countries, etc.

Anonymous links cannot save and continue across multiple computers. With mailing links, if you start a survey at one computer, you can then finish at another. However, the anonymous link, will only save to that particular survey (as long as the save and continue feature is enabled) to the computer that started the survey. When the system isn't recording who is responding, any new computer will start a new survey.

Anonymous surveys cannot guarantee one response per link. When the system doesn't know who is responding, it has no control over respondents forwarding their links to their friends. As long as it's a new computer, it's a new response.

Perhaps this feels restrictive, but one thing to keep in mind is that the links are designed to be anonymous and not capture identifying information. All of these features are possible within Qualtrics, but have been removed in order to ensure anonymity of responses. Use these links wisely.

Qualtrics provides users with a Cross Tabulation feature that can be used to find relationships between two variables. When making a Cross Tabulation, at the bottom of the table, there is a Chi-Squared statistic and a p-value. This statistic helps you measure the probability of seeing the distributions we do in the data, assuming the two variables of interest are independent. A low p-value (less that 0.05) suggests there is a relationship between the two variables. The Chi-Squared is best used to find a relationship between two categorical variables like any Likert scale question and many demographic questions. If you want to find a relationship between two continuous variables, consider using R² in Excel, SPSS, or another statistics package.

Many Qualtrics users want to evaluate how several variables affect a single response variable. The Cross Tabulation would be an exhausting process since each variable would have to be evaluated separately. The better way to do this would be to run a linear regression or logistic regression analysis in a statistics software package. This would allow you to see which variables influence your response and which ones do not. This will also help you make forecasts for your response variable. The data may have to be transformed to account for categorical variables, but this is not hard in most software packages.

Determining how data will appear before results are gathered is critical to creating a good survey, but it can be difficult to do. Clicking through the survey numerous times to simulate responses is very tedious, and data sets of five or ten often aren’t useful in determining how a survey should be structured.

The Test Survey feature has resolved this issue. It allows the user in the Qualtrics tool to collect ‘fake’ responses in order to see how data will appear. This component will allow a full data set to appear to the user before they ever distribute the link.

In the Edit Survey tab, click on ‘Test Survey’ in the Advanced Options drop-down menu (just to the right of the Launch Survey icon). Decide on the number of iterations you would like, then click ‘Start Test.’ When completed, the user can access the randomized data in the View Results tab.