Quantitative Panels
What's on this page
About Quantitative Panels
A quantitative panel is a great option when you’re looking to recruit certain demographics from commonly sampled segments of the population for a specific survey or study. You can build your quantitative panel demographics from scratch, or use a customizable template to get started.
Creating a Quantitative Panel
The high-level process to launch a quantitative panel includes:
- Defining your target audience
- Specifying your study’s needs
- Confirming the panel’s feasibility and cost
- Launching the panel
Attention: Do not edit your survey while panel response collection is in progress.
Go to the Online panel section of the Distributions tab.
Click Create a panel.
If you’d like to define your panel’s demographics yourself, select Start a Quantitative panel from scratch.
Alternatively, to use a preconfigured template, select any of the Quantitative panels and review the panel’s makeup on the righthand side of the screen.
Qtip: You can customize the template in the steps that follow. These customizations include editing and removing quotas, including or excluding screening criteria values, or removing screening questions entirely.
Click Next.
Qtip: You can click Save and exit at any step in the setup to save your panel as a draft. You can select Save as template to save it as a template instead.
Name your panel.
Configure your Panel details, which includes the following:
Number of responses: The number of responses you’d like collected for your research. You can collect between 50 to 10,000 responses per panel.
Qtip: When determining this number, you should consider the analyses you want to perform on your collected data. For a 95% confidence interval ±5, the general guidance is ~350 responses per cut of the dataset. For example, if you plan to review male versus female data separately, you would want 700 responses (350 per group). See the Sample Size Calculator for help determining your sample size.
Qtip: You can collect as low as 5 responses, but Qualtrics does not enforce quotas or allow data reconciliation when you collect less than 50 responses.
- Country or region: The country or region the respondent should be from. The available targeting criteria varies by country.
Language: The language the respondent should speak.
Attention: When using a panel template, the quotas and questions are based on the template’s default country and language. If you modify the country or language, please make the appropriate adjustments to the remaining quota and question criteria to ensure the panel is valid and feasible.
- Survey length (LOI): The amount of time it takes to complete your survey. This should be between 1 to 29 minutes.
Qtip: If the LOI is 30 minutes or over, we recommend launching a Qualitative panel instead.
- Incidence rate (IR): The target percentage of panelists who complete the survey after starting it. See Calculating Incidence Rate for help determining your incidence rate.
- Time to collect responses: The number of days the panel will be open for responses. This should be between 1 to 20 days.
Attention: Changing any of these fields except for panel name may affect your panel’s cost and feasibility. See Panel Cost and Feasibility for more information.
Choose a targeting criteria, which determines the demographic makeup of your panel and adds screening questions to your survey. The values of the screening questions are stored in your survey dataset as embedded data fields.
After choosing targeting criteria, you can customize which values will pass your screening question (for example, if you’d only like to collect survey responses from a certain region of your country). If you are using numeric targeting criteria such as Age or Household Income, click Custom ranges to enable custom ranges. For more information on custom ranges, see “Custom Ranges for Targeting Criteria” at the end of this section.
Qtip: The question text that is shown to respondents is displayed below this menu.
Click Add Quotas to set limits on each demographic. You must have at least one quota set in your panel.
Qtip: Items with quotas can only sum to 30 across all criteria (e.g., Male 49%, Female 51% would be 2 items with quotas). There is no limit to items without quotas.
Adjust the percentage makeup for each demographic group. Your quotas must equal 100%.
Qtip: Respondents who are screened-out by the screener questions are not recorded and they do not count as auditable responses.
Qtip: Click Remove Quotas if you want to ask the question of your panelists but not require certain quota targets to be met.
To add another demographic screener, click Add targeting criteria and repeat the above steps to define the demographics. The number of targeting criteria you can add and what criteria is available is dependent on your selected country.
Qtip: To remove targeting criteria, uncheck the box next to the demographic name.
If needed, you can revert all changes you’ve made to the panel by clicking Reset all panel details and targeting criteria.
Changes to your panel’s demographics may affect your panel’s cost and feasibility, which are displayed in the upper right corner. See Panel Cost and Feasibility for more information.
Qtip: If your panel cost is too high to purchase in the platform, you have a few options:
- If you’re a standard user, your division may be out of subscription credits. Reach out to a Brand Administrator.
- If you’re a Brand Admin, see steps on allocating subscription credits. Otherwise, reach out to your Account Executive to purchase more.
If you’re not using subscription credits, you can reach out to your Qualtrics Research Services representative.
Click Next.
Review your panel makeup and cost.
If you’d like to change your panel details or targeting criteria, click Edit to return to the setup screen.
When you’re ready to continue with your panel, click Next.
Select when you’d like to pay and fill out your payment details. See Payment Options for more information.
Accept the Terms of Services.
Click Invoice Panel or Purchase Panel.
Qtip: At this time, you can only have 5 panels open at a time.
Custom Ranges for Targeting Criteria
If you are using numeric targeting criteria such as Age or Household Income, you can set custom ranges for the values that will pass your screening question.
Example: If you want to target respondents in specific income brackets, you can create custom ranges for the “Household Income” criterion.
After enabling custom ranges, enter your desired values in the text boxes. Values must be a whole number, and the values within ranges cannot overlap.
Click Add range to add a range, or click the trash can icon to delete a range. You can have a maximum of 6 ranges for each criterion.
When you use custom ranges for your criteria, special embedded data fields will be added to your survey flow when your panel launches.
Qtip: It is possible to have gaps in your ranges, as respondents who fit into the gaps will be filtered out.
Qtip: For the Age criterion values must be between 18-99, and each range must include at least 5 years. For the Household Income criterion values must be between $0-999,999, and reach range must include at least $10,000.
Data Reconciliation
Attention: Do not delete a response in Data & Analysis if you plan on replacing it via reconciliation.
Reconciliation is the process of identifying low quality panel responses (such as fraudulent and spam responses) and rejecting them from your dataset. After responses have been identified for reconciliation, Qualtrics will try to replace them (or refund them if they cannot be replaced).
Attention: Data reconciliation is not available in the following cases:
- If the panel has less than 50 participants: These are considered test launches where the user is experimenting with survey/panel design and the collected responses are not expected to be high quality.
- If the panel is closed: Once the panel is closed, panel participants have already been paid for their time and all financial accounting records are closed.
- If you’ve reconciled 25% of your purchased number of completes: You can reconcile up to 25% of the purchased responses for a panel. You can do these in smaller batches (for example, reconcile 10% one day and then 15% another day), but the total number cannot exceed 25%. If you are reconciling responses in multiple batches, then you must wait for one batch to finish before you start the next one.
- If you’ve already deleted the responses from Data & Analysis: Responses are deleted from Data & Analysis after they’ve been reconciled. If a response has already been deleted, it can no longer be reconciled and replaced (since its record no longer exists to match up with our panel providers).
- If the fielding window has closed and you’ve already reconciled once since closure: You can reconcile responses only 1 time after your field window has closed but before the panel closes entirely.
To reconcile your responses:
In the Online panel section of the Distributions tab, click on the specific panel you’d like to reconcile responses for.
Go to the Reconciliation tab. In addition to letting you annotate responses, this tab displays the following information:
- Responses you can remove right now: The number of responses that have been collected and can be removed via reconciliation.
- Responses removed: The number of responses that have already been removed via reconciliation.
- Total responses you can remove: The total number of responses you are able to remove via reconciliation.
If desired, click Annotate responses to identify low-quality and fraudulent responses in your data. An overall score of 0 or greater will be provided to represent the quality of your responses, which you can use to determine which responses you want to reconcile. For more information on the type of responses that will be flagged and the embedded data fields that will be added to your data set, see “Annotated Responses” at the end of this section.
Qtip: It may take up to 90 minutes for annotation to finish. You can navigate to other parts of the platform during this time. You will receive a notification when your responses have been annotated and are ready for reconciliation.
Attention: This option is only available if the Annotate responses AI feature is enabled. See Artificial Intelligence (AI) Administration for more information.
Qtip: Non-English responses are automatically translated to English before scoring. Score accuracy for non-English content has not yet been validated.
Once your responses have been annotated, click Reconcile responses.
Click Download panel data and reconciliation template.
Open the file that contains your responses in a spreadsheet editor on your computer. For any responses you’d like to reconcile, copy the Response ID. Open the template file in a spreadsheet editor.
Qtip: Check out Qualtrics’ response quality features to help you identify low quality responses you’d like to reconcile.
In the Response ID column (A), add the response IDs you copied in step 5. Each response ID should be on its own row in the file.
In the Reason for removal column (B), add a code that represents why you’re removing the response. Use the key at the top of the file as a guide. These codes are optional unless you’re removing a sizable portion of your responses.
Qtip: If you’ve reconciled 10% or more of your total purchased responses, you must provide a removal reason for all ResponseIDs. This threshold applies across all reconciliation requests for the panel.
Save your file as a CSV.
In Qualtrics, click Select a CSV file and choose the file you saved in step 9.
Your file will be evaluated for any errors. If there are any errors, correct the issue described in the error message and then upload a new file.
Click Next.
Review the responses you’d like to reconcile. If you need to make any changes, you can click the Upload a different file link which will return you to the previous screen to upload a different file.
Once you’re sure you have the responses you’d like to reconcile, click Remove responses.
Click Confirm.
Once you confirm the action, the following will happen:
- Reconciled responses are deleted from the dataset and are removed from the “total responses” count for the panel.
Any quotas that were affected by the reconciled responses will be decremented.
Qtip: This only affects quotas that are created as part of the panel setup. Any custom quotas will need to be manually corrected by deleting the response and choosing to decrement quotas.
- New participants will be recruited to the panel to fill the reconciled responses. If participants cannot be found, then you will be refunded the responses that could not be filled.
Annotated Responses
Qtip: While we have guardrails in place and are continually refining our products, artificial intelligence may at times generate output that is inaccurate, incomplete, or outdated. Prior to using any output from Qualtrics' AI features, you must review the output for accuracy and ensure that it is suitable for your use case. Output from Qualtrics' AI features is not a substitute for human review or professional guidance.
AI-powered annotation will identify unsatisfactory responses. Unsatisfactory responses can be flagged for the following reasons:
- Duplicates: Duplicate IPs + demographics (where available), as well as duplicate open-ended text.
- Speeding: Unusually fast completion at the page or survey level based on device type and layout.
- Open-ended quality: Gibberish, nonsensical, repetitive, or multiple brief non-insightful answers.
- Straightlining: Unlikely uniform patterns in matrices relative to what’s expected for that question.
- Multi-select overselection: Selecting all/most options where unexpected across at least 2 items.
- Bot/digital fingerprinting: When available, suspicious repeat respondents identified based on metadata.
Responses will be assigned a quality score of 0 or greater based on the above flags, where a lower score signifies a lower risk, and a higher score signifies a higher risk. This value can be used to assess relative risk of individual responses to help determine if you want to reconcile those responses. 2 new fields will be added to the response data:
Q_QualityScore: The quality score assigned to each response.
Attention: A lower quality score does not guarantee a higher quality response.
Q_Quality_TIMESTAMP: The time the annotation ran and the quality score was applied.
Qtip: If you re-run your annotations, the Q_Quality_TIMESTAMP field will be updated with the time of the most recent annotation.
You can use the following ranges to interpret quality scores:
- Not suspicious: Scores less than 15
- Suspicious: Scores between 15-29
- Fraudulent: Scores above 30
Loosening Quotas
Qtip: This option is only available for quantitative panels.
While fielding your panel, it is possible that some quota groups may struggle to fill completely. If this happens, you can loosen your minimum quota requirements while your panel is still active, enabling you to collect all responses in time to complete your project. Loosening allows you to relax a quota target to a more achievable range. This is accomplished by choosing a percentage to expand the quota by and calculating a new acceptable target range above and below the original target.
Loosening is applied at the individual quota level (e.g., age or region), allowing you to protect your most critical variables while relaxing others.
Example: You are running a nationwide survey for men aged 18-24. Your country is split into 4 separate regions, so you set a 25% quota for each region. With a few days left in the study, you notice that the quota for one region filled very quickly, while the other ones have struggled to fill, and you worry you may not receive enough responses to complete your study. By loosening the quota for region, you can now collect responses from panelists whose region demographic falls outside of the initial targets. Because of this quota flexibility, more panelists are able to qualify for the survey. As a result, the final dataset may be more representative of some regions than others, but the target number of responses has been met.
Attention: Loosening or opening your quotas is irreversible. You can further loosen your quotas if needed, but they cannot be tightened.
Select your panel while it is still collecting responses.
- In the Quotas tab, find the quota you’d like to loosen and click Loosen quotas.
Review your quotas and how they’re performing against the original targets.
- Choose how much you’d like to loosen quotas, from 5% to 25%.
- Alternatively, you can click Open Quota to remove this demographic quota. This demographic metric will still be tracked in responses, but will no longer be used to determine respondent eligibility for the survey.
- Review the projected impact which shows the new target range of responses for each quota.
- Click Confirm.
Panel Cost and Feasibility
Panel cost and feasibility are 2 important metrics when setting up your panel. This section covers both metrics and how customizing your panel can impact these values.
Qtip: If Qualtrics is unable to collect the number of responses specified in your panel, you will be refunded the cost of any uncollected responses, minus associated processing fees.
- Cost: Cost is calculated based on your survey length, incidence rate, time to collect responses, selected template, and number of responses. Changing these values will change the panel price.
Feasibility: Feasibility measures the confidence that your panel will collect all desired responses. Your panel can either be Feasible or Not Feasible depending on your panel setup.
Qtip: You will not be able to launch your panel if it is “Not Feasible.”
In general, the more complicated your panel audience, the higher your panel will cost and the less likely you will be to find respondents who match your panel criteria. Generally, the following increase the cost and lower the feasibility:
- The more responses purchased (e.g., 5000 responses is harder than 500)
- The longer the survey is (e.g., 50 min survey is harder than a 5 min survey)
- The lower the incidence rate is (e.g., 20% IR is harder than 80% IR)
- The shorter the time to collect is (e.g., 1 day is harder than 10 days)
- The more specific your targeting criteria is (e.g., full time employees in agriculture are harder to find than general US citizens)
Qtip: You can switch between a cost summary in credits or USD.
Calculating Incidence Rate
The incidence rate is the percentage of people from a target population that meet specific eligibility criteria to participate in your survey and who complete the survey. This section will cover how to estimate an incidence rate for your survey and how the incidence rate for your survey is calculated once you start collecting responses.
Estimating Incidence Rates
Estimating an incidence rate is important because it directly impacts the cost and feasibility of your panel. Follow the below steps to calculate your incidence rate:
Example: Let’s say we want to survey the general American population who uses smartphones. We’ll continue using this example population for the remaining steps.
Example: Since we’re conducting a survey about smartphone usage, our eligibility criteria includes owning a smartphone and being within a specific age bracket (18-35, 35-54, and 55+). Additionally, we are only interested in surveying people who live in the United States.
Example: Based on independent research, around 85% of Americans own smartphones. Additionally, US census data shows around 78% of Americans are above 18 years old.
Example: If 85% of Americans own smartphones and 78% of Americans are above 18 years old, then 66% of Americans are both above 18 and own a smartphone.
Example: Since our estimated incidence rate is 66%, we use 60% as our target incidence rate.
Additionally, the following survey design aspects can impact your survey’s incidence rate:
Survey length: The longer a survey is, the more likely a respondent is to drop off, which decreases your survey’s incidence rate.
Attention: An attention check is a question designed to make sure the survey respondent is paying attention. Too many attention checks or too strict of attention checks can reduce the number of people who qualify for your survey.
- Complex quotas: When you have a lot of quotas, certain combinations may fill up faster than others. As a result, eligible participants may not qualify because those quotas have already filled up.
Qtip: If you are completely unsure of what your estimated incidence rate should be, enter 40%. When creating your panel, start with a small panel of 50 respondents. After the panel is complete, you will have an observed incidence rate, and you can create a new panel with the new incidence rate.
Observed Incidence Rates
As panelists respond to your survey, your survey’s incidence rate will be calculated to give you an overview of how respondents are progressing through your survey.
Incidence rate is calculated using the following formula:
Incidence Rate = (Number of Complete Survey Responses) / (Number of Valid Clicks)
In the above formula, the variables are defined as:
- Number of Complete Survey Responses: The number of qualified responses that have completed the full survey and have been validated by quality measures.
- Number of Valid Clicks: The total number of times the survey has been clicked, which is calculated by adding together the following values:
- Complete Responses: The number of qualified responses that have completed the full survey and have been validated by quality measures.
- Over-Quota Responses: The number of responses that were removed from the survey because they fulfilled a quota that was already full.
- Screened Out Responses: The number of responses that were removed from the survey due to failing the screening criteria.
- Dropped Responses: The number of responses that ended due to 24 hours of inactivity. A dropped respondent can be the result of survey programming errors, redirection issues, or a respondent losing interest in a survey and not completing it.
Qtip: Responses that are removed due to data quality issues aren’t included in incidence rate calculations.
Creating Custom Screening Questions
If you want to add a screener that is not included in the current targeting options, you can set one up manually in your survey project. Follow the below steps before publishing your survey and setting up the online panel. Keep in mind that more screener questions can cause your incidence rate to be lower (since there are more chances for respondents to be disqualified).
https://api.spectrumsurveys.com/surveydone?st=18&transaction_id=${e://Field/transaction_id}Warning: When you select Do NOT record survey response, screened out responses are not recorded, and there is no way to retrieve them. Only select this option if you are absolutely sure you do not want to record screened out responses at all.
That's great! Thank you for your feedback!
Thank you for your feedback!