Online Panels
What's on this page
About Online Panels
Qtip: Panels are only available for survey project types.
A panel is a group of participants who have been recruited to provide feedback on a research topic. You can define targeting criteria and launch your research study to this audience. For example, let’s say you’re launching a product designed for pet owners and would like to gather feedback on the early concepts. You can use a panel to send your survey to only pet owners, enabling you to quickly collect accurate and relevant feedback without the hassle of finding these individuals yourself.
Qtip: Only panels costing up to 15,000 USD can be purchased online using the steps on this page. For panels exceeding this price, please reach out to your Account Executive or Account Services.
Attention: This feature is not available for customers in the FedRAMP environment.
Video Walkthrough
Need help getting started with online panels? Check out this 2-minute video walkthrough:
Preparing a Survey for Online Panels
Follow the below steps to get your survey ready for distribution via an online panel:
Qtip: You do not need to ask any template questions in your survey to capture respondent demographic data. If you select “age” in your template, then you will automatically collect age values in your dataset. If you want to ask the question in a particular way or with particular answer options, you can re-ask it in your survey with the desired format. This custom version will not contribute to quotas for panel eligibility.
Creating a Quantitative Panel
A quantitative panel is a great option when you’re looking to recruit certain demographics from commonly sampled segments of the population for a specific survey or study. You can build your quantitative panel demographics from scratch, or use a customizable template to get started.
The high-level process to launch a panel includes:
- Defining your target audience
- Specifying your study’s needs
- Confirming the panel’s feasibility and cost
- Launching the panel
Attention: Do not edit your survey while panel response collection is in progress.
Qtip: At this time, you can only have 5 panels open at a time.
Creating a Qualitative Panel
Qualitative panels are often used for studies that go beyond a simple survey, such as those requiring screen-sharing or multiple rounds of feedback. The purpose of qualitative panels is to find participants comfortable engaging in these exercises while giving detailed feedback. The participants will not be anonymous, coming with a profile of information attached as embedded data.
Example: In UX research, you might require screen, camera, or microphone-sharing. Qualitative research can help you find candidates willing to perform these actions while giving actionable feedback.
Qtip: Due to the greater study demands, qualitative panels tend to have higher incentives provided by researchers themselves, as opposed to the panel provider. Qualtrics Support cannot advise on incentive pricing. We recommend looking for resources with best practices tailored to your industry and type of study. For example, Respondent has a detailed article on incentive pricing.
Qtip: Qualitative panels tend to be better for low volume studies of 1 to 50 participants.
Inviting and Managing Qualitative Panel Candidates
As suitable participants are identified for your qualitative panel, you can choose which candidates you want to officially invite to participate in the study. Look through the candidates, and invite the ones you want to participate. Only invited participants receive the survey. From there, you can choose to either approve or reject the participants’ responses.
Qtip: If you pause the panel for 7 days, the panel is automatically closed, and unreviewed responses are auto-approved.
Attention: It’s important to review responses as they come in. Only approved panelists receive an incentive, but any responses left “awaiting approval” will be automatically moved to “approved” and incentivized once the panel is closed.
Qtip: If a rejected or approved participant did something you feel should disqualify them from future studies, you can report them. Keep in mind that you should only report participants in severe cases. You can only report a limit of 5 total participants across your entire Qualtrics brand (i.e., license).
When you report a participant, you must provide details why.
Synthetic Panels
Synthetic panels are powered by a first party proprietary AI model developed here at Qualtrics. Our synthetic panel is trained on thousands of responses from a variety of demographic backgrounds in order to more accurately predict how certain populations would respond to a survey.
To learn more, see our Synthetic Panels page.
User-Made Templates
If you have access to panels, you can create panel templates, allowing you to easily reuse the same panel configuration across multiple distributions.
When creating a new panel, you can choose to save that panel as a template.
Payment Options
You can enter payment details to pay for your online panel immediately, or you can choose to generate an invoice and pay later.
Pay Now
If you’re paying immediately and not generating an invoice, there are 2 available payment methods:
US bank account: Set up an electronic transfer of funds from your bank account. This method uses Automated Clearing House (ACH) payments. For more information, see ACH Payments.
Attention: ACH payments can take 1-3 business days to process. Your panel will not launch until the payment is processed.
Qtip: This payment method uses Stripe for verification.
- Card: Pay using a credit or debit card.
Pay Later
In the Pay later tab, enter the information of the company or person that you would like to send the invoice to.
Once generated, a PDF of the invoice and a private payment URL will be sent to the email address that was entered. This email will come from Stripe. Click Pay this invoice to pay your invoice on Stripe. You can also share the payment URL with another party.
Enter your payment information on Stripe and select Pay. Your panel will be launched after successful payment.
Qtip: The panel will not launch until the invoice is paid. See Managing Panels for more information.
Attention: The invoice will expire after 30 days if unpaid.
Qtip: Invoices can only be paid through the Stripe payment link provided by the panel tool. Online Panels is not supported by other Qualtrics invoice and payment systems. If a payment is mistakenly made through another system, email invoices@qualtrics.com for help.
Subscription Credits
Subscription credits – also known as Edge Audiences – offer a way for you to set a budget for panels without having to get separate panel purchases budgeted and approved. Instead, you buy all the credits you need at once, to use at your leisure. You can also add more credits as needed.
You can only purchase subscription credits from your Account Executive.
Subscription credits can be used to pay for any type of panel, human or synthetic. They are the only way to purchase synthetic panels.
Qtip: Subscription credits are shared across the organization (brand). Credits cannot be allocated to individual users, but you can always limit which users have access to online panels with the Access Online Panels user permission.
Qtip: Brand Administrators can track their organization’s subscription credit consumption using the subscription credit admin report.
How Panels Change Your Survey
Once you purchase a panel, your survey will be automatically updated to make sure your panel’s data is collected correctly. These changes happen in your project’s survey flow.
After creating a panel, you’ll notice the following additions to your survey flow:
- A new branch called “Online Samples API – All Fields” at the top of the survey flow, which includes the panel embedded data used to track respondent data from the panel supplier.
- A new security branch at the bottom of the survey flow to ensure panel security.
- When the panel is closed, the security branch will automatically be deleted from the survey flow. The “All Fields” branch will remain so that you can use the embedded data fields in your analysis.
If you launch another panel on the same survey, any new fields will be added to the existing “All Fields” branch. The security branch will also be re-added while the new panel is active.
Qtip: Existing fields are not duplicated. Only new fields are added.
Attention: Do not edit these branches or their contents at any time. Do not edit your survey while panel response collection is in progress.
Managing Panels
After you’ve requested the panel and your payment is processed, your panel will be created. You can then click on the panel to view more information about the panel’s status, expected end time, number of collected responses, and more.
After selecting the panel, a window will open that displays the following information about your panel:
- Survey: The name of the survey being used for this panel.
- Status: The panel’s status. Possible statuses include:
Draft (not launched): The panel has not yet been created because it is not fully set up.
Qtip: If your panel is a draft, click the Edit panel button to edit it.
- Collecting: The panel is currently open and fielding new responses.
- Collection ended: The panel is complete and has stopped collecting responses.
Paused: The panel is not currently collecting responses but can be resumed.
Qtip: Pausing a panel does not extend the fielding window. The panel will stop collecting responses 20 days after launching.
Collection ended: The panel was not able to collect all responses within 20 days and has stopped collecting responses.
Qtip: This status may also appear if the panel supplier has closed response collection. In that case, a banner will appear telling you to contact technical support.
- Closed: The panel is closed and no new responses can be collected via the panel.
Awaiting payment: The invoice for the panel has been sent but has not been paid yet. The expiration date and link to the invoice are provided underneath the status.
- Payment failed: The invoice link has expired or the payment submitted from the invoice link has failed. You can click the invoice link underneath the status to re-submit the payment, or you can delete the panel and start over with a new invoice.
- Launch failed: The payment was successful, but the panel failed to launch. We recommend reaching out to Qualtrics Technical Support.
- Launch time: The date and time that the panel was created.
Expected end time: The requested date and time for the panel to be completed (i.e., when the goal number of responses is collected).
Qtip: While the expected end time is the target, it is possible that the panel may not complete within this timeframe. The panel will remain open for a maximum of 20 days to attempt to collect all responses.
- Fielding window: The number of days between the launch time and expected end date.
- Distribution Id: The ID of the distribution. This value will also appear on your purchase receipt.
- Total responses: The number of responses collected and the goal number of responses.
- Survey length (LOI): The observed survey length measured in the live panel compared to the predicted survey length.
Incidence rate (IR): The observed incidence rate measured in the live panel compared to the predicted incidence rate.
Qtip: If your observed LOI and IR are performing worse than predicted and your panel collection looks unlikely to collect all responses, we recommend pausing response collection and creating a new panel with updated parameters to get new costs and feasibility.
Attention: If your panel did not successfully fulfill all responses, we recommend pausing the panel and configuring a new one with the observed IR and LOI to recalculate the cost and feasibility. Loosening quota requirements may also help improve feasibility.
Qtip: LOI, IR, and Quotas are settings available for quantitative panels, and do not apply to qualitative panels.
Quotas Tab
You can view the quotas set for your panel even before response collection is complete. Using this method, you can see the progress that’s being made to meet your demographic quotas.
- Count: How many responses fitting this quota have been collected so far.
- Target: How many responses should be collected for the quota by the end of the panel.
Data Reconciliation
Attention: Do not delete a response in Data & Analysis if you plan on replacing it via reconciliation.
Reconciliation is the process of identifying low quality panel responses (such as fraudulent and spam responses) and rejecting them from your dataset. After responses have been identified for reconciliation, Qualtrics will try to replace them (or refund them if they cannot be replaced).
Attention: Data reconciliation is not available in the following cases:
- If the panel has less than 50 participants: These are considered test launches where the user is experimenting with survey/panel design and the collected responses are not expected to be high quality.
- If the panel is closed: Once the panel is closed, panel participants have already been paid for their time and all financial accounting records are closed.
- If you’ve reconciled 25% of your purchased number of completes: You can reconcile up to 25% of the purchased responses for a panel. You can do these in smaller batches (for example, reconcile 10% one day and then 15% another day), but the total number cannot exceed 25%. If you are reconciling responses in multiple batches, then you must wait for one batch to finish before you start the next one.
- If you’ve already deleted the responses from Data & Analysis: Responses are deleted from Data & Analysis after they’ve been reconciled. If a response has already been deleted, it can no longer be reconciled and replaced (since its record no longer exists to match up with our panel providers).
- If the fielding window has closed and you’ve already reconciled once since closure: You can reconcile responses only 1 time after your field window has closed but before the panel closes entirely.
To reconcile your responses:
Once you confirm the action, the following will happen:
- Reconciled responses are deleted from the dataset and are removed from the “total responses” count for the panel.
Any quotas that were affected by the reconciled responses will be decremented.
Qtip: This only affects quotas that are created as part of the panel setup. Any custom quotas will need to be manually corrected by deleting the response and choosing to decrement quotas.
- New participants will be recruited to the panel to fill the reconciled responses. If participants cannot be found, then you will be refunded the responses that could not be filled.
Pausing a Panel
Qtip: Panels cannot be manually closed; instead, they automatically close 7 days after response collection has ended (i.e., 27 days after the panel was launched).
You can pause response collection for an open panel at any time. This does not immediately close the panel; it simply stops new responses from being collected. If the fielding window is still open, you can resume response collection for the panel.
Attention: Once a panel is closed, you cannot edit it and it will no longer collect any new responses. If there are any unfulfilled responses, a prorated refund will automatically be issued to the credit card used to purchase the panel.
Panel Cost and Feasibility
Panel cost and feasibility are 2 important metrics when setting up your panel. This section covers both metrics and how customizing your panel can impact these values.
Qtip: If Qualtrics is unable to collect the number of responses specified in your panel, you will be refunded the cost of any uncollected responses.
- Cost: Cost is calculated based on your survey length, incidence rate, time to collect responses, selected template, and number of responses. Changing these values will change the panel price.
Feasibility: Feasibility measures the confidence that your panel will collect all desired responses. Your panel can either be Feasible or Not Feasible depending on your panel setup.
Qtip: You will not be able to launch your panel if it is “Not Feasible.”
In general, the more complicated your panel audience, the higher your panel will cost and the less likely you will be to find respondents who match your panel criteria. Generally, the following increase the cost and lower the feasibility:
- The more responses purchased (e.g., 5000 responses is harder than 500)
- The longer the survey is (e.g., 50 min survey is harder than a 5 min survey)
- The lower the incidence rate is (e.g., 20% IR is harder than 80% IR)
- The shorter the time to collect is (e.g., 1 day is harder than 10 days)
- The more specific your targeting criteria is (e.g., full time employees in agriculture are harder to find than general US citizens)
Calculating Incidence Rate
The incidence rate is the percentage of people from a target population that meet specific eligibility criteria to participate in your survey and who complete the survey. This section will cover how to estimate an incidence rate for your survey and how the incidence rate for your survey is calculated once you start collecting responses.
Estimating Incidence Rates
Estimating an incidence rate is important because it directly impacts the cost and feasibility of your panel. Follow the below steps to calculate your incidence rate:
Additionally, the following survey design aspects can impact your survey’s incidence rate:
Survey length: The longer a survey is, the more likely a respondent is to drop off, which decreases your survey’s incidence rate.
Attention: An attention check is a question designed to make sure the survey respondent is paying attention. Too many attention checks or too strict of attention checks can reduce the number of people who qualify for your survey.
- Complex quotas: When you have a lot of quotas, certain combinations may fill up faster than others. As a result, eligible participants may not qualify because those quotas have already filled up.
Qtip: If you are completely unsure of what your estimated incidence rate should be, enter 40%. When creating your panel, start with a small panel of 50 respondents. After the panel is complete, you will have an observed incidence rate, and you can create a new panel with the new incidence rate.
Observed Incidence Rates
As panelists respond to your survey, your survey’s incidence rate will be calculated to give you an overview of how respondents are progressing through your survey.
Incidence rate is calculated using the following formula:
Incidence Rate = (Number of Complete Survey Responses) / (Number of Valid Clicks)
In the above formula, the variables are defined as:
- Number of Complete Survey Responses: The number of qualified responses that have completed the full survey and have been validated by quality measures.
- Number of Valid Clicks: The total number of times the survey has been clicked, which is calculated by adding together the following values:
- Complete Responses: The number of qualified responses that have completed the full survey and have been validated by quality measures.
- Over-Quota Responses: The number of responses that were removed from the survey because they fulfilled a quota that was already full.
- Screened Out Responses: The number of responses that were removed from the survey due to failing the screening criteria.
- Dropped Responses: The number of responses that ended due to 24 hours of inactivity. A dropped respondent can be the result of survey programming errors, redirection issues, or a respondent losing interest in a survey and not completing it.
Qtip: Responses that are removed due to data quality issues aren’t included in incidence rate calculations.
Creating Custom Screening Questions
If you want to add a screener that is not included in the current targeting options, you can set one up manually in your survey project. Follow the below steps before publishing your survey and setting up the online panel. Keep in mind that more screener questions can cause your incidence rate to be lower (since there are more chances for respondents to be disqualified).
Viewing Panel Data
Qtip: Panel data is not scrubbed for anonymity or sensitive data. This service can be provided by the Qualtrics Research Services team for an additional cost. If you do not have a research representative, reach out to your Account Executive.
You can view your responses by navigating to the Data & Analysis tab of your survey. Responses to screening questions are added to survey responses as embedded data. The embedded data field names use the naming scheme: “_os_api_variable.”
Example: If asking respondents for their age, then you will have an embedded data field called “_os_api_age.” If asking respondents for their gender, then you will have an embedded data field called “_os_api_gender.”
There are a few additional embedded data fields for every survey that uses online panels:
- _os_api_panel_id: The ID of the distribution. This will match your distribution ID for the panel project.
- _os_api_vendor: Third party panel vendor used to source the panel respondents.
- _os_api_participant_id: This field is only available for qualitative panels. This is a unique ID for each participant.
- gc: This stands for “good complete.” A value of “1” means the respondent fully completed the survey and will be paid for their time.
- psid: A unique value that the panel supplier assigns to the research participant to anonymously identify them.
- transaction_id: A unique value that the panel supplier assigns to associate the survey session with the research participant.
- pureSpectrumRedirectUrl: Research participants are redirected back to the panel supplier portal on completion of the survey. This field is only included for templated (quantitative) panels.
- pureSpectrumSignatureValue: A security measure to avoid fraud. This field is only included for templated (quantitative) panels.
Additional fields are available depending on the demographic criteria used for your panel:
- _os_api_gender: The respondent’s gender. Possible values include:
- Male
- Female
- Prefer not to say
- _os_api_hispanic: Indicates if the respondent is Hispanic. This field is only included for templated (quantitative) panels. Possible values include:
- Yes
- No
- _os_api_ethnicity: The respondent’s ethnicity. Possible values include:
- White
- Black or African American
- Asian
- Native Hawaiian or Other Pacific Islander
- American Indian or Alaska Native
- Other
- _os_api_region: The respondent’s region in the United States. This field is only included for templated (quantitative) panels. Possible values include:
- Northeast
- Midwest
- South
- West
- _os_api_age: The age range of the participant. Values include:
- 18-24
- 25-34
- 35-44
- 45-54
- 55-64
- 65+
- _os_api_industry: The industry the participant is in. This field is only available for qualitative panels and for quantitative (templated) panels that are relevant, such as job-focused ones.
- _os_api_job_function: The type of job the panelist has. This field is only available for qualitative panels.
- _os_api_company_size: The size of the company the panelist works for. This field is only available for qualitative panels.
That's great! Thank you for your feedback!
Thank you for your feedback!