Significance Testing in Simple Charts & Simple Tables (CX)

About Significance Testing in Simple Charts & Simple Tables

Dashboards can help you understand whether the differences you see over time or between groups are statistically significant, and therefore worthy of driving important business decisions. For example, you may have found yourself asking the following:

• Did the NPS really go up this month, or is it a small change that’s just noise in the data?
• Does the Midwest group actually have higher satisfaction scores than the West group?
• Which of my five segments had higher or lower than typical scores on this metric?

With significance testing in simple line charts and simple tables, you can discover what data changes matter most.

Available Widgets and Metrics

Significance testing is currently available in the following widgets, with the following parameters. We’ll go over these in more detail in the following sections.

Qtip: Only proportional custom metrics with a single field as the divisor can be used for significance testing. Proportional custom metrics follow the general format of (A + B) / C where A, B, and C are different data fields. A / B also works, since there is just a single field as the divisor. When creating custom metrics for this purpose, you can only use counts, as shown below. There cannot be static number in the equation; e.g., (A + 5) / B will not work.

Significance Testing in Simple Charts

Simple line charts can flag the significantly high or low values of a change over time for NPS scores and top / bottom boxes. Each time period is compared against each time period previous when determining significance of a change.

1. Add a simple chart widget.
2. Make it a line graph.
   
3. Click Add Metric.
4. Choose one of the eligible metrics for simple chart.
5. Click Set X Axis Dimension.
   
6. Add a date field of your choice.
   Qtip: You can only have one x-axis dimension for this to work, and it must be a date field.
7. Click on your metric.
   
8. Go to Options.
9. Select Enable Significance Testing.
10. Select your Confidence Interval.

Significance Testing in Simple Tables

Significance testing can be performed on a simple table when using NPS or top / bottom box metrics broken out over an x-axis dimension of your choice.

1. Add a simple table widget.
2. Choose one of the eligible metrics for simple table.
   
3. Click Set Row Dimension and add a field of your choice.
4. Click on your metric.
   
5. Go to Options.
6. Select Enable Significance Testing.
7. Determine what you want your significance testing should be based off of. You have the following options:
   • Identify particularly high or low values: Most common selection for data that doesn’t involve time or dates.
     Example: Is Brazil’s score higher than other South American countries?
   • Compare each time period to previous: Most common selection for data that involves time or dates. Because of this, the option only appears if your x axis dimension is a date field.
     Example: Is this month’s overall score higher than last month’s?
   • Compare each value to each other value: This option is less commonly used, but is useful when comparing one baseline group to all other groups.
     Example: Is Brazil’s score higher than Venezuela’s score? Is Brazil’s score higher than Colombia’s? Is Venezuela’s higher than Colombia’s?
     Qtip: This option only appears if you have 20 or fewer data groups to compare. For example, let’s say you’ve collected data for 3 quarters. A simple table broken out by day cannot compare each day to each other day because there are 90 days of data. In contrast, if the Simple Table were broken out by quarter, there are only 3 quarters to compare to each other.
8. Select your Confidence Interval.

Understanding Significance Testing

The Confidence Interval indicates how confident you would like to be that the results generated through the analysis match the general population. Higher confidence levels raise the threshold for a difference to be considered statistically significant, meaning only the clearest differences will be marked as such.

Once you have enabled significance testing, you might notice upwards and downwards arrows in your widget. These arrows indicate statistically significant values.

You can hover over an arrow to determine why the value is considered significant, and what the confidence interval of that test was.

Technical Notes on Significance Testing

When comparing one NPS score to another, regardless of chart type or type of comparison (e.g., over time), the following process is used:

1. Create a new column of data that recodes NPS scores in the following fashion:
   • Promoters = 100
   • Neutrals = 0
   • Detractors = -100
2. Run a two-tailed Welch’s independent samples t-test.

When comparing one top box score to another, regardless of chart type or type of comparison (e.g., over time), the following process is used:

1. Create a new column of data that recodes raw scores into TRUE or FALSE, depending on whether they meet the top box criteria.
2. Run a two-tailed z-test for difference in two proportions.