Response Weighting

About Response Weighting

Imagine that you have a product used equally as often by men and women. But when your customer feedback on this product returns, 70% of your respondents were female, and 30% were male. If you were to present these results as collected, they would skew in favor of the women’s answers, even though they are only 50% of the target demographic.

Response weighting allows you to change the weights of variable fields so the data in your Reports tab will reflect targeted demographics.

Adding Weights in the New UI

1. Click Create new scheme.
   
2. Select the type of weighting you want to use.
   
   Your options include:
   • Raked weighting
   • Cell based weighting
   • Static weighting
     Qtip: Check out the section on Raked vs. Interlocked Weighting for more information about these weighting types.
3. Click Add a variable.
   
4. Choose the variable you want to weight.
5. Adjust the weighting for the variable by typing the target values into the appropriate boxes.
6. To add additional variables to your weighting, click Add a variable and then repeat steps 4 and 5.
   Qtip: You can add variables until the Cardinality Limit of your weighting reaches 5,000. See the Weight Reports for more information about Cardinality Limits.
7. If you want to remove a variable from your weighting, click the trash can icon next to that variable.
   Qtip: You can also click the X next to the variable name above your weighting scheme to remove the variable.
   
8. As you add weights, the Weight report will adjust to show you your Cardinality Limit. You can add variables to your weighting scheme until the Cardinality Limit reaches 5,000. See Weight Reports for more details.
9. Click Next. After clicking Next, your weights will calculate which may take up to a couple minutes. The Weight Report will also update to show you information about your weighting scheme.
10. Review your weights to make sure everything is correct.
11. Read the Weight Report for more information about your weighting scheme.
12. If you need to make changes, click Back.
13. Once you’re finished, click Save weights.

Qtip: After saving your weights, you can edit them by clicking the three dots and then Edit.

Weighting Multiple Variables: Raked vs. Interlocked

Raked Weighting

Raked Weighting takes a simplistic approach to multiple variables. You can add up to two variables at a time, and each one is configured separately. This type of weighting is best if you’re looking at non-overlapping variables.

For example, if you wish to weight both Gender and Spanish/Hispanic/Latino identity, you determine your desired percentages of each Gender, then of each identity. The software will then calculate the weights of overlaps (for example, Hispanic Women or Latino Men) for you.

Qtip: The value under Calculated Weight will change to yellow when your weight is invalid. In this example, our total weighted percentage was 120%, exceeding the maximum of 100%.

Interlocked Weighting

Interlocked Weighting is useful if you’re interested in targeting overlapping variables.  For example, if you have a larger audience of Spanish Women than Spanish Men, you can adjust these percentages accordingly.

There are two different types of Interlocked weighting schemes:

• Cell Based: Define values in the Distribution column. These are measured in percentages adding up to 100%. When you define these values, the system automatically defines the Weight column for you.
  Qtip: This weighting scheme is called Dynamic Interlocked weighting in the legacy UI.
  Qtip: If you are not as familiar with weights, we recommend using this option.
• Static: Edit the values in the Weight column. These numbers are what the data in a given category will be multiplied by once collected. 1 means no weight is applied.

Weight Reports

Once you create your weighting scheme, the Weight report will update, showing you key pieces of information about your weighting.

1. Input Variables: These are the variables that are being used in the weighting scheme.
   
2. Cardinality Limit: Cardinality is calculated by multiplying the number of categories for each of your variables together. The Cardinality Limit cannot pass 5,000. The pie chart in this section will adjust to show you how close you are to reaching the Cardinality Limit.
   Example: For example, let’s say you are weighting three variables with 10, 5, and 7 categories respectively. Your Cardinality limit is then 350 since that is the product of the number of categories (10 x 5 x 7 = 350).
3. Sample Balance: The Sample balance is a measurement of weighting efficiency during rake weighting convergence. As more variables are added, the balance can be driven down when weights are extreme (having very large or very small values), impacting overall weighting efficiency.
   
4. Weight Efficiency: This section shows you how efficient your weighting scheme is.
   • Minimum weight: The minimum weight. This is represented as a multiplier instead of a percentage out of 100.
   • Maximum weight: The maximum weight. This is represented as a multiplier instead of a percentage out of 100.
   • Weights less than 1: Of all the characteristics you’re weighting, this is the number of characteristics that have a multiplier less than 1.
   • Weights greater than 1:  Of all the characteristics you’re weighting, this is the number of characteristics that have a multiplier greater than 1.
   • Characteristics for minimum weight X: X varies based on what your minimum weight is. Each combination of variables has a corresponding weight. Characteristics for minimum weight are the variable combinations corresponding to the minimum weight.
     Example: Let’s say we are weighting education, and we gave “Professional degree (JD, MD)” a targeted weight of 2%, and every other education level a targeted weight above 10%. “Professional degree (JD, MD)” would be our characteristic for minimum weight.
   • Characteristics for maximum weight X: X varies based on what your maximum weight is. Each combination of variables has a corresponding weight. Characteristics for maximum weight are the variable combinations corresponding to the maximum weight.
5. Quality: This metric shows if any of your variables have missing values (i.e. no respondents selected a specific response value).

Exporting & Importing Weights

After creating your weighting scheme, you can export the weights so you can easily use them again in the future. This section will go over importing and exporting weighting schemes.

Exporting Weights

1. While editing an interlocked weighting scheme, click Action.
   
2. Click Import / Export Weights.
3. Select Export current weights to download a CSV file with your weighting.
   

Importing Weights

1. While editing an interlocked weighting scheme, click Action.
   
2. Click Import / Export Weights.
3. Click Choose file and select the weighting file on your computer.
   
4. Select Submit.

Adding Weights in the Legacy UI

Qtip: The new weighting experience offers an better UX and improvements to weighting functionality. If you’d like to use the old weighting UI, click Use the previous version and follow the steps below for creating your weights.

After you’ve collected some data, you’ll be ready to start weighting your variables. You can find this feature under the Data & Analysis tab when you click the Weighting section. Note that any question or answer choice that has not collected data cannot be weighted.

1. Click on the green Add a Variable button.
2. Pick a question, Embedded Data, or Survey Metadata to weight.
3. On the top right, choose either Raked or Interlocked Weighting.
   Qtip: If you are weighting just one variable, Raked and Interlocked Weighting work the same way. However, if you are looking to weight more than one variable, navigate to our Weighting Multiple Variables section to determine which type of weighting is best for you.
4. Specify the desired percentage for each field. These must add up to 100.
5. Click Save Weights.

The Raked Weighting page requires you navigate to the Raked Weights table before you click Save Weights.

Note that the Interlocked Weighting page puts the Save Weights button in the top right corner instead.

You cannot export weights in the legacy UI. However, you can export and import Interlocked weights using the new UI. See Importing & Exporting Weights for more details.

Reports

Response weighting is automatically applied to your Reports tab under the Results section. You can choose to turn this on or off either at the Global Level (for a whole report), or at the Visualization Level (for a single graph or table).

Turning off Weighting for a Visualization

1. Select the visualization.
2. Hover over the top left corner so that the blue scale icon appears.
3. Click this icon so it turns gray.

Turning off Weighting for a Report

1. Go to Report Options.
2. Click Global Options.
3. Deselect Use weighted metrics.
4. Click Save Settings.

Compatible Variables

Any variable with a finite number of answer choices can be supported by response weighting. This includes:

• All questions with a Single or Multiple Answer option
• Rank Order
• Pick, Group, and Rank
• Hot Spots and Heat Maps with defined regions
• Start Date or End Date by Day, Week, Month, Quarter, or Year
• Finished status
• Response Type
• Respondent Language
• Embedded Data set to Multi-Value Text Set, Number Set, or Text Set type
• Custom Tags
• Bucketed Variables

Variables where the answer choices are infinite – such as Text Entry fields – are not compatible.