Sentiment Analysis

About Sentiment Analysis

Qualtrics will assign a Very Negative, Negative, Neutral, Positive, Very positive or Mixed sentiment to a text response as soon as it is loaded in Text iQ. This sentiment is based on the language in the response and the question text. The Text iQ model is trained on a large and diverse set of real experience data to substantially increase classification quality and minimize uncertainties when the sentiment is not expressed as strongly. Once assigned, each sentiment comes with a numeric score, called the sentiment score.

The Very Negative and Very Positive sentiment labels have been implemented to indicate the strongest sentiment, helping you to focus on the most critical feedback in each comment. These sentiment measures can be used to filter your data or be displayed in visualizations in the Reports tab.

Sentiment Functionality

For most Text iQ analysis, the question text is not relevant to the sentiment of the answer (e.g. “Why did you give us that score?”). However, occasionally the framing of the question will imply a sentiment for the answer (e.g. “What’s 1 thing we could improve about our product?”). In these cases, Text iQ needs both the question and the answer to make an accurate prediction of sentiment. For example, if the question was, “What do you love about our company?” then unless the answer is explicitly negative (e.g. “Nothing, we hate your company”) then the sentiment of the response is probably positive, even if the answer text on its own appears neutral (e.g. “pricing”). This Sentiment Analysis update incorporates the question text, if available, in the sentiment analysis so that the predicted sentiment is as accurate as possible.

This model also now takes sentiment edits into account in order to learn how to better assign sentiments in the future. However, please note that these edits do not immediately affect the sentiment assignment of projects in your account. Instead, these edits are periodically analyzed across all Qualtrics users and used to improve the sentiment analysis model as a whole.

Overall Sentiment vs. Topic Sentiment

Topic Sentiment is the sentiment score of a particular topic in your text response. Responses can have multiple Topic Sentiment scores as each topic is assigned its own score. Topic sentiment is calculated by isolating the part of the response that matches your query, and determining what the sentiment of that individual part is.

Overall Sentiment is the sentiment score for a given response. Every response analyzed in Text iQ will have only 1 overall sentiment score.

Topic Sentiment is the sentiment score of a particular topic in your text response. Responses can have multiple Topic Sentiment scores as each topic is assigned its own score. For topics that do not have queries, topic sentiment will default to overall sentiment

Overall sentiment, overall sentiment score, topic-level sentiment, and topic-level sentiment score are included in response exports. Overall sentiment is labelled just as “Sentiment” and “Sentiment-Score.” Topic-level is labelled “Topic Sentiment Label” and “Topic Sentiment Score.” Topic-level fields can have multiple values (to account for multiple topics) and are listed as Topic: Sentiment. E.g., Dessert: Negative. Topics are also included in response exports.

Viewing and Changing Sentiment

1. Make sure edit mode is opened. You will see a Return to summary button in the upper-right corner.
   
2. Navigate to your responses. There will be an icon to the left of each response indicating sentiment. This represents the sentiment score.
   • 2 red down arrows: Very Negative.
   • 1 red down arrow: Negative.
   • Overlapping gray diamonds: Mixed.
   • 1 green up arrow: Positive.
   • 2 green up arrows: Very Positive.
   • Gray diamond: Neutral.
3. Click the edit comment icon.
4. Navigate to the Comment Overall Sentiment section.
   
   Qtip: Note that you are editing the Comment Overall Sentiment of the entire response.
5. Click a new sentiment score.
6. Click Save.

Topic Sentiment

When you select Topic Sentiment, you can specify the sentiment for each topic tagged on the response. This is especially helpful if your overall sentiment is mixed, or if the intensity of the sentiment varies for each topic the respondent covered.

You will be able to select sentiment and sentiment score under each topic name.

Under Other topics in your model, you also have the option to add additional topic tags and sentiments directly on the response. Click the suggested topics or create your own topic with the blue plus ( + ) sign.

Filtering by Sentiment

1. Go to Data & Analysis, Results, or Reports.
   
2. Click Add Filter.
3. Select Embedded Data.
4. Select the sentiment you are interested in. They will be labeled by question number.
5. Choose which type of sentiment you are interested in filtering by (Negative, Neutral, Positive, Very Negative, or Very Positive).

Sentiment Score

In addition to labelling a comment positive, negative, mixed, or neutral, there will be a numeric value assigned to each sentiment.

The Sentiment Score is a -2 to +2 score for the sentiment of a comment. For example, a very positive response such as “I love Love LOVE YOUR COMPANY!” would score a +2. A very negative response, such as, “I UTTERLY DESPISE your company!” would get a -2. A neutral comment receives a score of 0.

The sentiment score can be found as a field in the Data & Analysis and in the Reports tabs. Like most text analysis variables, you will find them under the embedded data category.

Sentiment Model Updates

Over time, Qualtrics makes improvements and releases minor version updates to our sentiment model. We conduct intensive quality assurance on these updates to ensure consistent or improved sentiment prediction accuracy.

When a new model version is released, it will only be used to analyze new comments in Text iQ. Comments previously analyzed by Text iQ will not be updated with the release of a new model.

When applying Text iQ to copies of projects or dashboards, it is possible to see some differences in sentiment predictions for historical responses. This is because at the time the responses in the original project or dashboard were analyzed, a previous version of our sentiment model was used. In the copied project or dashboard, sentiment predictions are recalculated using our current model.

Multi-Language Sentiment Analysis

Analyzing sentiment in other languages is available to all users who have Sentiment Analysis, and thus all Advanced Text clients.

The Qualtrics sentiment model is a transformer-based cross-lingual model that leverages the latest deep learning technologies. Our model has been optimized for English, French, Spanish, German, Korean, Italian, Dutch, Japanese, Portuguese, Traditional Chinese, Simplified Chinese, Polish, Swedish, Thai, Indonesian and Russian. Our model can make sentiment predictions for all languages, but can not guarantee the same level of quality as predictions for optimized languages.

You can read more about how we developed our Sentiment Analysis model on this blog post.