Response Weighting

About Response Weighting

Imagine that you have a product used equally as often by men and women. But when your customer feedback on this product returns, 70% of your respondents were female, and 30% were male. If you were to present these results as collected, they would skew in favor of the women’s answers, even though they are only 50% of the target demographic.

Response weighting allows you to change the weights of variable fields so the data in your Reports tab will reflect targeted demographics. Wave based weighting can also be used to apply unique weights over various time periods or specific categories, which can be useful for long-running programs that require multiple weighting configurations.

To get started, go to the Data & Analysis tab of your survey and select Weighting.

Adding Weights

1. Click Create new scheme.
   
2. Select the type of weighting you want to use.
   
   Your options include:
   • Raked weighting
   • Cell based weighting
   • Static weighting
     Qtip: Check out the section on Raked vs. Interlocked Weighting for more information about these weighting types.
   Qtip: If you want to use waved weighting, see Wave Based Weighting.
3. Click Add a variable.
   
4. Choose the variable you want to weight.
5. Adjust the weighting for the variable by typing the target values into the appropriate boxes.
   Qtip: This weighting feature has been updated to display 6 digits of precision. This does not alter any existing percentages or weight calculations.
6. To add additional variables to your weighting, click Add a variable and then repeat steps 4 and 5.
   Qtip: See Cardinality for more information, including limits.
7. If you want to remove a variable from your weighting, click the trash can icon next to that variable.
   Qtip: You can also click the X next to the variable name above your weighting scheme to remove the variable.
   
8. As you add weights, the Weight report will adjust to show you your
   Qtip: See Cardinality Limit.
9. Click Next. After clicking Next, your weights will calculate which may take up to a couple minutes. The Weight Report will also update to show you information about your weighting scheme.
10. Review your weights to make sure everything is correct.
11. Read the Weight Report for more information about your weighting scheme.
12. If you need to make changes, click Back.
13. Once you’re finished, click Save weights.

Qtip: After saving your weights, you can edit them by clicking the three dots and then Edit.

Weighting Multiple Variables: Raked vs. Interlocked

Raked Weighting

Raked Weighting (also known as Rim target) takes a simplistic approach to multiple variables. You can add up to two variables at a time, and each one is configured separately. This type of weighting is best if you’re looking at non-overlapping variables.

For example, if you wish to weight both Gender and Spanish/Hispanic/Latino identity, you determine your desired percentages of each Gender, then of each identity. The software will then calculate the weights of overlaps (for example, Hispanic Women or Latino Men) for you.

Qtip: The value under Calculated Weight will change to yellow when your weight is invalid. In this example, our total weighted percentage was 120%, exceeding the maximum of 100%.

Interlocked Weighting

Interlocked Weighting is useful if you’re interested in targeting overlapping variables.  For example, if you have a larger audience of Spanish Women than Spanish Men, you can adjust these percentages accordingly.

There are two different types of Interlocked weighting schemes:

• Cell Based: Also known as dynamic interlocked response. Define values in the Distribution column. These are measured in percentages adding up to 100%. When you define these values, the system automatically defines the Weight column for you.
  Qtip: If you are not as familiar with weights, we recommend using this option.
• Static: Also known as interlocked response. Edit the values in the Weight column. These numbers are what the data in a given category will be multiplied by once collected. 1 means no weight is applied.

Wave Based Weighting

Wave based weighting allows you to apply unique weights over various time periods or specific categories.

1. Click Create new scheme.
   
2. Enter a name for your new weight scheme.
   
3. Select the type of weighting you want to use.
   • Raked weighting
   • Cell based weighting
   • Static weighting
   Qtip: See Raked vs. Interlocked Weighting for more information about these weighting types.
4. Check Data needs to be weighted as waves.
5. Select a time zone from the dropdown. If you choose to break out your waves by time periods, these time periods will be set based on your selected time zone.
6. Click Next.
7. If you want to weight only specific responses, add a filter. Only responses that meet your conditions will be weighted. The number of qualified responses for weighting will update as you adjust your filter.
   
   Qtip: If you want to weight all your responses, do not add a filter.
8. Click Next.
9. Select how you would like to configure your waves. Your options include:
   • Date range: Waves are defined using a date field, such as Start Date or End Date, and waves are broken out by different time periods.
   • Categorical: Waves are created based on non-date fields, such as survey questions or survey metadata.

   

10. Select a field from the dropdown to define your wave.
    Attention: Dates within a field group won’t appear in the date field dropdown. These fields must be mapped as dates outside of a field group for them to appear.
    Qtip: For categorical waves, all fields types are supported except for Geometry, Date Time, Enumberable Value or Enumerable Value Set.
11. If you are configuring your waves with a date range, select ​​a time period for the wave to be broken out by.
12. Select Configure wave.
13. Select the wave you want to create a weighting configuration for.
    
    Qtip: Click Rename waves if you would like to give your waves different names.
14. Click Next.
15. Decide if you want to Apply the current wave configuration to more waves. You can select specific waves or select All remaining waves. If you leave this option unchecked, your weighting configuration will only apply to the selected wave.
    Qtip: If desired, you can rename your wave by typing a new name into the Wave name text box.
16. Click Add a variable.
    
17. Choose the variable you want to weight.
18. Adjust the weighting for the variable by typing the target values into the appropriate boxes.
    
    Qtip: You can also import weights using a CSV or TSV, which is useful If you are adjusting the weighting of many variables. See Exporting & Importing Weights for more information.
19. To add additional variables to your weighting, click Add a variable and then repeat steps 16 and 17.
20. If you want to remove a variable from your weighting, click the X next to the variable name above your weighting scheme to remove the variable.
21. As you add weights, the weight report will adjust to show you your Cardinality Limit.
22. Click Next. Your weights will now be calculated, which may take up to a couple minutes. The weight report will also update to show you information about your weighting scheme.
23. Review your weights to make sure everything is correct.
    
24. Read the Weight report for more information about your weighting scheme.
25. If you need to make changes, click Back.
26. Once you’re finished, click Save weights.
    Attention: If you’re creating weights in CX Dashboards, you need to Publish your weighting scheme after you save it.
    
    Qtip: To edit weights that have been saved, click the three horizontal dots in the top-right corner for non-waved schemes or next to each wave for waved schemes. Then, select Edit from the menu.

Weight Reports

Once you create your weighting scheme, the Weight report will update, showing you key pieces of information about your weighting.

1. Input Variables: These are the variables that are being used in the weighting scheme.
   
2. Cardinality Limit: Cardinality refers to the number of unique weight values that will be generated during the weighting process. Learn more about Cardinality.
3. Sample Balance: The Sample balance is a measurement of weighting efficiency during rake weighting convergence. As more variables are added, the balance can be driven down when weights are extreme (having very large or very small values), impacting overall weighting efficiency.
   
4. Weight Efficiency: This section shows you how efficient your weighting scheme is.
   • Minimum weight: The minimum weight. This is represented as a multiplier instead of a percentage out of 100.
   • Maximum weight: The maximum weight. This is represented as a multiplier instead of a percentage out of 100.
   • Weights less than 1: Of all the characteristics you’re weighting, this is the number of characteristics that have a multiplier less than 1.
   • Weights greater than 1:  Of all the characteristics you’re weighting, this is the number of characteristics that have a multiplier greater than 1.
   • Characteristics for minimum weight X: X varies based on what your minimum weight is. Each combination of variables has a corresponding weight. Characteristics for minimum weight are the variable combinations corresponding to the minimum weight.
     Example: Let’s say we are weighting education, and we gave “Professional degree (JD, MD)” a targeted weight of 2%, and every other education level a targeted weight above 10%. “Professional degree (JD, MD)” would be our characteristic for minimum weight.
   • Characteristics for maximum weight X: X varies based on what your maximum weight is. Each combination of variables has a corresponding weight. Characteristics for maximum weight are the variable combinations corresponding to the maximum weight.
5. Quality: This metric shows if any of your variables have missing values (i.e. no respondents selected a specific response value).

Cardinality

Cardinality refers to the number of unique weight values that will be generated during the weighting process. The goal of a cardinality calculation is to count the unique weights that will actually be generated, not the number that could theoretically exist.

Cardinality can be less than or equal to the number of responses you’ve collected, but never greater.

Adding a variable to your weighting scheme doesn’t always increase the cardinality; collecting data that reflects new combinations does. Again, this is because cardinality is based on actual data as opposed to possible data.

For Interlocked (Cell-Based) Weighting

Interlocked weighting creates a weight for each unique combination of variable categories that exists in your data. Cardinality is thus based on present data collected, not all possible options that could eventually exist.

For Raked (IPF) Weighting

Raked weighting uses Iterative Proportional Fitting (IPF) to assign weights to individual responses based on marginal distributions. As with interlocked, the cardinality is based on the unique combinations of weighting variables present in the actual data. This count cannot exceed the number of responses you collected.

Exporting & Importing Weights

After creating your weighting scheme, you can export the weights so you can easily use them again in the future. This section will go over importing and exporting weighting schemes.

Exporting Weights

1. While editing an interlocked weighting scheme, click Action.
   
2. Click Import / Export Weights.
3. Select Export current weights to download a CSV file with your weighting.
   

Importing Weights

1. While editing an interlocked weighting scheme, click Action.
   
2. Click Import / Export Weights.
3. Click Choose file and select the weighting file on your computer.
   
4. Select Submit.

Reports

Response weighting is automatically applied to the reports in the Results tab. You can choose to turn this on or off either at the Global Level (for a whole report), or at the Visualization Level (for a single graph or table).

Turning off Weighting for a Visualization

1. Select the visualization.
2. Hover over the top left corner so that the blue scale icon appears.
3. Click this icon so it turns gray.

Turning off Weighting for a Report

1. Go to Report Options.
2. Click Global Options.
3. Deselect Use weighted metrics.
4. Click Save Settings.

Compatible Variables

Any variable with a finite number of answer choices can be supported by response weighting. This includes:

• Single answer multiple choice questions
• Rank Order
• Pick, Group, and Rank
• Hot Spots and Heat Maps with defined regions
• Start Date or End Date by Day, Week, Month, Quarter, or Year
• Finished status
• Response Type
• Respondent Language
• Embedded Data set to Number Set or Text Set type
• Custom Tags
• Bucketed Variables

Variables where the answer choices are infinite – such as Text Entry fields – are not compatible.

Projects Where Responses Can Be Weighted

Response weighting is only available in a few different types of projects:

• Survey projects
• XM Solutions
  Qtip: Response weighting is not available in Pricing Studies (Gabor Granger projects).
• Imported data projects