Loading...
  • Customer Experience
    Customer Experience
  • Employee Experience
    Employee Experience
  • Brand Experience
    Brand Experience
  • Product Experience
    Product Experience
  • Core XM
    Core XM

Idea Screening XM Solution (PX)

What's on This Page:


Was this helpful?


This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

The feedback you submit here is used only to help improve this page.

That’s great! Thank you for your feedback!

Thank you for your feedback!


About the Idea Screening XM Solution

The Idea Screening XM Solution allows you to screen a set of ideas to discover which ideas resonate the most with your respondents. This solution targets early-stage innovation efforts to help you understand which ideas and product lines have the most potential before investing further in their development.

Methodology

Developed by Qualtrics ProductXM scientists, Idea Screening is an automated XM Solution that can help organizations understand their customers’ product preferences. The Idea Screening setup asks for a high-level description to provide context around your set of ideas, then asks for a simple text description of each idea.

Gut Reaction Question

MaxDiff is a powerful methodology that is used to assess the relative preference of your respondents for your set of ideas. The first part of the survey will ask respondents to give their gut reaction to each idea. They are given a short period of time to review the idea, then a short period of time to answer questions about their gut feeling about the idea. If your respondent doesn’t answer within five seconds, the survey will automatically advance to the next question.Question about Poufs gauging gut reaction

Qtip: Respondents will be given a descriptive text question that explains the timing behavior before being presented with the gut reaction questions. A timing question will automatically be included in the survey, so no extra setup is required on your part!Introduction question giving directions to the respondent

Qtip: If more than five ideas are configured in the setup, additional MaxDiff questions and analysis will be included.

Detailed Idea Screening Questions

Next, respondents will be asked to give a more detailed breakdown of how much they like or dislike each idea, and how new or different each idea is.

Anchored MaxDiff Questions

MaxDiff is a powerful methodology for assessing relative preference of respondents across the set of ideas. When you include eight or more ideas, the survey template will automatically include Advanced MaxDiff methodology, which will add anchored MaxDiff questions. In this case, the user will be asked to choose their most and least preferred idea, and they’ll be asked a follow up question assessing idea importance.

Creating an Idea Screening Survey

  1. Navigate to the Projects page by clicking the XM logo or by clicking Projects in the top-right.Projects page with Create new project indicated
  2. Click Create new project.
  3. Navigate to the ProductXM tab.Navigating to the ProductXM page and clicking on the Idea Screening XM Solution below
  4. Under Product Research, select the Idea Screening XM Solution.
  5. Name your project.Naming your project and clicking Get Started
  6. Click Get Started.
  7. Enter an introduction to explain the survey to your participants.Guided setup for adding an introduction for your participants and adding product ideas
  8. Define your ideas. We recommend including a one to two sentence description for each idea.
    Qtip: If you would like to include images with each of your ideas, select Include Images. Once this option is selected, you’ll have the option to attach an image when you define each idea. Just click Browse to include your image!Option to include images with each idea
    Qtip: Click Add Another to add additional ideas.Add Another indicated to add more ideas
  9. Select additional demographics to include as questions in the survey.Add Demographics section and the Finish Button indicated
    Qtip: If you are collecting demographics data, we recommend including 5 or fewer demographic questions for a reasonable survey length.
    Qtip: The selected demographics questions will be added in a block at the end of your survey. According to our ProductXM Scientists, it is best practice for demographics questions to come at the end of the survey, rather than at the beginning. When people have to answer specific information about themselves upfront, they feel less anonymous. This can result in people answering less truthfully or providing more socially desirable answers.Demographics section in the Survey Editor Demographics block at the end of the Survey Flow
  10. Click Finish.
Attention: We recommend that you make sure to collect the suggested number of responses (at the least) in order to yield statistically valid results. This will be viewable in the configuration of your Idea Screening project. In this example, the suggested sample size is 190 respondents; you may have a different suggested sample size in your Idea Screening project depending on your survey setup. For more details, visit our MaxDiff Analysis White Paper.Suggested number of respondents window detailed; found on the right side of the guided setup

Adding Screening Questions

When gathering feedback on your ideas, our ProductXM Scientists recommend that you look to the broadest possible pool of potential customers. You can screen your ideas with a simple question like, “Have you used this product category in the past xx months?” This will ensure that you’re surveying a wide audience, but you’re still making sure that your responses will be from potential customers. This section will detail how to add relevant screening questions to your survey in order to determine whether your respondents meet the right criteria to take part in the study.

Example: Let’s say that you are using Idea Screening to gauge ideas for a new podcast. In this case, you may not want to include the opinion of someone who never listens to podcasts.
Warning: It’s important to make edits to the survey before you distribute it to customers. If possible, avoid making edits to a survey actively collecting data, or you may risk invalidating your data and changing the premade reports that come with this solution. See Testing / Editing an Active Survey.
Qtip: For more information on how to handle and/or troubleshoot screened-out responses, see Screen-Out Management.
  1. You will be navigated to the Survey Editor.
  2. Click the drop down arrow next to the Screening Questions block. The Guidance pathway on the right side will provide additional information about screening questions.Creating a question under the Screening Block
  3. Select Create a New Question.
  4. Create your screening question(s). See Creating Questions for more details.A screening question has been written in the Screening block
  5. Navigate to the Survey Flow. You’ll be using branch logic in the Survey Flow to screen out participants that don’t meet the criteria in your screening question(s).
  6. Click Add Below on your Screening Questions block.Add Below and Branch are indicated in the Survey Flow under the Screening Block
  7. Select Branch.
  8. Click Add a Condition.Adding a condition on the Branch to create logic
  9. Define what your logic will be based on as Question, since you’re screening out respondents based on your screening question(s).Creating Logic on the Branch
  10. Select the question you would like your logic to be based on. In this case, you should select the question you’re using to screen respondents out with.
  11. Select the choice that you would like to screen out.
    Example: In this example, we would like to screen out respondents who don’t buy leather. After the Screening Questions Block, there is a branch that will end the survey if the respondent says that they only own 0-1 leather goods. This will ensure that the opinions of people who don’t own leather will not be recorded in the Idea Screening survey.End of survey element created
    Attention: Make sure that you’re careful about how you order your blocks and branches. If your elements are incorrectly ordered, they may not work properly. We recommend following the format of the examples provided!
  12. Set the drop down to Is Selected.
  13. Click OK.
  14. Click Add a New Element Here.
  15. Select End of Survey. See End of Survey Element for more details, as well as for additional Customization Options.Selecting an End of Survey Element under the branch
  16. Click Save Flow.
  17. Click Done.Publishing your survey
  18. Publish your survey.
  19. Review your survey. You can either click the Preview button at the top of your survey, or select Preview Survey in the Guidance section to preview your survey. See Preview Survey for more details.Previewing the survey after publishing

Creating a Customer Contact List

Qtip: We highly recommend having a list of customers who you are sending the survey to prepared and on-hand before you create your project.
Qtip: If you would like to include Embedded Data in your contact list, make sure that you also add your Embedded Data to your project’s Survey Flow.

File Requirements

Contact lists should always be a file saved in one of the following formats, even if you’re using an example file we provided:

  1. CSV (Comma Separated Values) with UTF-8 encoding
  2. TSV (Tab Separated Values)
Qtip: TSV is preferred if  the customer data contains anything other than numbers and the English alphabet. See CSV / TSV Upload Issues for troubleshooting help.

Note that the maximum file size for a contact list upload is 100 MB.

Information to Include in the File

The List of Available Contact Fields has information on how to format certain required fields, such as Email, FirstName, and LastName.

Qtip: Feel free to use this Example Contact File (CSV).

Distributing Your Idea Screening Survey

Once you are finished with survey edits and your contact list is ready, it’s time to distribute your survey.

  1. Make sure you have published any survey edits.
  2. Click Select Distribution in the Guidance or Distributions in the upper-left.Navigating to Distributions
  3. Select a distribution type. See Distributions for more details.Selecting a specific distribution type

Dashboards

Our XM Scientists have developed a special dashboard for this XM Solution. The Idea Screening dashboard has two built-in pages: the Summary dashboard and the Idea Breakdown dashboard. These dashboards will give an overview of your results, and they will be ready to share as soon as you have collected data.

Qtip: The dashboards will not generate charts and tables until you collect data. Try completing the survey through the preview button if you want to test what these dashboards look like with data. Remember you can always delete preview data later.

Summary

Summary page of the Idea Screening Dashboard

The Summary page shows the MaxDiff preference share data from the project, the average utility of each item, and rankings for each idea. Your ideas will be ranked according to their gut reaction, favorite idea, top 2 like/dislike, and top 2 new/different ideas.

Idea Screening Widgets

This section explains each of the built-in widget configurations available in the Idea Screening dashboard.

  • Preference Share: Measures the distribution of preferences for the tested product ideas. The preference share totals 100 and represents the likelihood that the idea is truly most preferred.Preference Share widget
  • Average Utility: The direct average calculation across respondents’ individual utility scores. These coefficients are derived from a Hierarchical Bayesian model and reflect the relative preference and importance of the ideas being tested.Average Utility widget
  • Count Analysis: Represents the percentage of times the idea was selected Most / Least preferred when it was shown.Count Analysis widget
  • Gut Reaction: Measure the initial reaction to a product idea. A larger percentage indicates that respondents “liked it a lot” or “liked it a little” in their initial reactions.Gut Reaction Rankings widget
  • Favorite Idea Ranking: Ranks ideas according to the percentage of respondents who marked it as their favorite out of the group of ideas.Favorite Idea Ranking widget
  • Top 2 Box – Liked a great deal or somewhat: Measures the approval of a given idea. A larger percentage indicates that respondents “Liked it a great deal” or “Liked it somewhat” in their overall approval.Top 2 Box - Like a great deal or somewhat widget
  • Top 2 Box – Extremely or very new/different: Measures the uniqueness of a given idea. A larger percentage indicates that respondents found the idea “Extremely” or “Very” new or different.Top 2 Box Extremely or very new/different widget
Qtip: Within Qualtrics, you can hover over the ( i ) symbol to view the tooltips for each of these widgets. The tooltips will have additional information about what each widget measures and how the data can be interpreted.Tooltip indicated on the Preference Share widget next to the title

Idea Breakdown

Idea Screening Idea Breakdown page in the dashboard

The Idea Breakdown page of the dashboard includes detailed breakdowns of your data. It has widgets that will show you the gut reaction, like/dislike, new/different, and favorite percentage breakdown for each idea.

Dashboard Editing

Qtip: The premade Idea Screening Dashboard was developed by our Qualtrics ProductXM scientists. We advise trying to keep your dashboard as similar to the original template as possible, and not making too many large-scale changes.

Check out these introductory pages to get started.

Qtip: Once you’ve uploaded all your dashboard users, you can easily send many users or send everyone their login credentials.