Sentiment Analysis

About Sentiment Analysis

Qualtrics will assign a Very Negative, Negative, Neutral, Positive, Very positive or Mixed sentiment to a text response as soon as it is loaded in Text iQ. This sentiment is based off of the language in the response, the question text itself, and edits you’ve made to your sentiment analysis. The Text iQ model is trained on a large and diverse set of real experience data to substantially increase classification quality and minimize uncertainties when the sentiment is not expressed as strongly. Once assigned, each sentiment comes with a numeric score, the sentiment score.

Qualtrics’ sentiment analysis is constantly improving. Our analysis tool periodically runs a batch analysis on all manual corrections made by users to improve the accuracy of the model, assess improvements, and correct mistakes that Text iQ is making in sentiment identification. The Very Negative and Very Positive sentiment labels have been implemented to indicate the strongest sentiment, helping you to focus on the most critical feedback in each comment.

These sentiment measures can be used to filter your data or be displayed in visualizations in the Reports tab.

Sentiment Functionality

For most Text iQ analysis, the question text is not relevant to the sentiment of the answer (e.g. “Why did you give us that score?”). However, occasionally the framing of the question will imply a sentiment for the answer (e.g. “What’s one thing we could improve about our product?”). In these cases, Text iQ needs both the question and the answer to make an accurate prediction of sentiment. For example, if the question was, “What do you love about our company?” then unless the answer is explicitly negative (e.g. “Nothing, we hate your company”) then the sentiment of the response is probably positive, even if the answer text on its own appears neutral (e.g. “pricing”). This Sentiment Analysis update incorporates the question text, if available, in the sentiment analysis so that the predicted sentiment is as accurate as possible.

This model also now takes sentiment edits into account in order to learn how to better assign sentiments in the future. However, please note that these edits do not immediately affect the sentiment assignment of projects in your account. Instead, these edits are periodically analyzed across all Qualtrics users and used to improve the sentiment analysis model as a whole.

Overall Sentiment vs. Topic Sentiment

When Qualtrics analyzes your text responses, there are two different sentiment scores that can be assigned. These scores are Overall Sentiment and Topic Sentiment. Understanding the difference between these two sentiment scores is important in understanding your text analysis data.

Overall Sentiment is the sentiment score for a given response. Every response analyzed in Text iQ will have only one overall sentiment score.

Topic Sentiment is the sentiment score of a particular topic in your text response. Responses can have multiple Topic Sentiment scores as each topic is assigned its own score.

Overall sentiment, overall sentiment score, topic-level sentiment, and topic-level sentiment score are included in response exports. Overall sentiment is labelled just as “Sentiment” and “Sentiment-Score.” Topic-level is labelled “Topic Sentiment Label” and “Topic Sentiment Score.” Topic-level fields can have multiple values (to account for multiple topics) and are listed as Topic: Sentiment. E.g., Dessert: Negative. Topics are also included in response exports.

Viewing and Changing Sentiment

1. Make sure comment mode is opened. You will see a blue checkmark in the upper-right, but you will not see the Add Widget button.
2. Navigate to your responses. There will be a diamond to the left of each indicating sentiment. This represents the sentiment score.
   • Solid Red: Very Negative.
   • Half Red: Negative.
   • Purple with a line in the middle: Mixed.
   • Half Blue: Positive.
   • Solid Blue: Very Positive.
   • White with Gray Outline: Neutral.
3. Click the edit comment button.
4. Navigate to the Comment Overall Sentiment section.
   Qtip: Note that you are editing the Comment Overall Sentiment of the entire response.
5. Select a sentiment score.

Topic Sentiment

When you select Topic Sentiment, you can specify the sentiment for each topic tagged on the response. This is especially helpful if your overall sentiment is mixed, or if the intensity of the sentiment varies for each topic the respondent covered.

You will be able to select sentiment and sentiment score under each topic name.

In the above screenshot, the respondent loves the food at the restaurant but believes that the staff is overworked, so we want to make sure the topic “All Food” has a positive sentiment, while the topic “Workforce” does not. The overall sentiment is mixed due to these two topic sentiments interacting with each other.

You also have the option to add additional topic tags and sentiments directly on the response by selecting suggested topics in the top right corner or by creating your own topic with the blue plus (+) sign.

Filtering by Sentiment

1. Inside the Data & Analysis tab or the Reports tab, click Add Filter.
2. Select Embedded Data.
3. Select the sentiment you are interested in. They will be labeled by question number.
4. Choose which type of sentiment you are interested in filtering by (Negative, Neutral, Positive, Very Negative, or Very Positive).

Sentiment Score

In addition to labelling a comment positive, negative, mixed, or neutral, there will be a numeric value assigned to each sentiment.

The Sentiment Score is a -2 to +2 score for the sentiment of a comment. For example, a very intense, positive response such as “I love Love LOVE YOUR COMPANY!” would score a +2. An intensely negative response, such as, “I UTTERLY DESPISE your company!” would get a -2. 0 is a neutral score.

The Sentiment Polarity is a 0 to 10 score for the polarity of the comment. This is a calculation of how mixed the comment is. For extremely positive and negative sentiment scores (like the previous examples), there’s usually a corresponding 0 polarity. But for a comment such as “I love your food, but your restaurant is so dirty,” you will get a more nuanced score.

The sentiment score and polarity can be found with the sentiment itself under the Data & Analysis and in the Reports tabs. Like most text analysis variables, you will find them under the embedded data category.

Qtip: Use Choose Columns to bring these variables into the Data & Analysis tab if you haven’t already!

Multi-Language Sentiment Analysis

Sentiment analysis is currently available in languages other than English, including: Chinese (Simplified), Chinese (Traditional), French, German, Italian, Japanese, Korean, Portuguese, Dutch, Thai, Indonesian, and Spanish.

Analyzing sentiment in other languages is available to all users who have Sentiment Analysis, and thus all Advanced Text clients.

When Multi-Language Sentiment Analysis is enabled, Qualtrics will detect the language of the response. So long as it is one of the languages listed above, this response will have sentiment analysis performed on it in its native language, capturing the nuances of the response in ways a translation may not. If this feature is not enabled, the sentiment analysis will not detect the language of the response and will assign sentiments as if it were written in English. Thus, when Multi-Language Sentiment Analysis is disabled, the sentiments assigned to responses not written in English are not as accurate.

FAQ