Your ultimate guide to concept testing

What is a concept testing survey?

It’s a tough job to manage the launch of your company’s products and services. You’re already facing several statistics that paint a grim picture of the reality of product launches:

• Only 55% of all product launches take place on schedule, and 45% of product launches are delayed by at least one month, according to a 2019 product manager survey from Gartner.
• More than 25% of total revenue and profits come from launching new successful products across industries, according to a McKinsey research survey.
• Preparing is the biggest problem when launching new products, according to a Harvard Business School report.

Under this pressure, you want to make sure your product connects with your customer before it goes to market. By doing the right product research, you’re checking that your assumptions about your customer are correct.

Product concept testing is an early-stage market research method that maximizes the odds of you launching a product or service that people want to buy. You explore the viability of a product or service with its target audience early on and improve its development from the feedback.

Free eBook: Introduction to product concept testing

What concept testing is not

Concept testing is not a replacement for the ideation phase of a development cycle. Instead, it’s a means of testing out concepts that have been developed to some extent, but need further definition. This kind of market research helps you explore responses to potential features and benefits that you envision going into the finished product or service, and to get a more detailed understanding of your target customer’s requirements.

Concept testing is also different from brand testing, advertising, or marketing campaigns – these promote the product or service when it’s near-complete or fully developed.

For more information on concept testing methods and making an effective concept testing survey, read our eBook: Introduction to Product Concept Testing

How concept testing fits into market research

Concept testing is an important aspect of market research, allowing the consumer to provide their feedback on whether or not a concept’s features and benefits fit their needs.

Concept testing is based on listening to the audience’s views and exploring how viable the concept is for them – without predefined parameters or expectations. It’s an open learning process where you can continue to engage with the same target audience and get their take on your product developments as they happen.

It’s also a means to exercise good Experience Management. At a product launch, your brand’s perceived value and the strength of your offering are directly linked to your profit and loss. Likewise, a disappointing loss can hurt internal staff morale, investor relationships, and future business ventures.

Special Event: The 3 steps of product concept testing

“Poor test-and-learn execution has been known to hobble a company’s fortunes for years to come.”

– Bain & Company

Concept testing is relatively cheap to do, and it provides feedback you can use to make the right decisions and prevent costly mistakes further along in the development process. It can also help steer the direction of later stages in research and development, because it gives you a clear picture of which criteria are being met and which ones need more attention. Here are 6 of the most appealing concept testing benefits.

1. Concept testing can be a cost-effective and flexible solution

You can set up a simple and quick survey if you want high-level feedback, or you can delve deeper to understand more detail.

Concept testing is also a great money-saving technique as it prevents launching a faulty concept. And in comparison with the cost of outsourcing to external market research agencies, research for an internal concept testing project is cheaper.

2. It helps you gain support for the concept at an early stage

If your plan for a product or service requires a lot of input from senior leaders or colleagues, providing them with a ‘first look’ at evidence can speed up the process and show the potential of the work they’ll be helping with. When a team is confident, there is more support and willingness to go the extra mile for you.

3. Concept testing can help you optimize the concept before launch

You can uncover or identify information that will have a true impact on your product development process decisions. A few examples are:

• The concept’s weaknesses (E.g. Is there unclear communication, low customer value, or an unmemorable product personality?)
• The customer’s price perceptions, buying preferences, how relevant they find the features and benefits, and how they might use it
• The concept’s status relative to its target market competition
• How well the concept fits with the brand

In the early stages, you can also test multiple concepts at the same time to see which works best at solving your target customer’s problem. If a product concept doesn’t work, invest in the ones that do make a difference. For later stages, you can use preliminary forecasts to estimate demand.

4. Concept testing acts as a quality-assurance check over time

Thanks to its low cost, concept testing can be iterated multiple times. This allows you to home in on multiple aspects of your concept and make sure all the requirements are covered. You can then repeat the analysis to see what effect your changes have made. For example – you changed the concept’s feature profile, and now you can investigate whether that has changed the value perception and potential revenues.

Concept testing programs also lay the foundations for future benchmarks. Service or product ideas that satisfy key criteria and land well with your audience can be used as barometers for new and/or similar concepts.

5. Research can help you build solid customer relationships

You can build brand and customer loyalty (find out more about customer loyalty) and increase your brand equity value, by including your potential customers in the concept’s design and development.

As a brand, you look transparent and open to innovation. It’s also a great way to show you value the opinions of your customers. In addition to this, you should include a broader sample to understand if and how your concept appeals to wider audiences — you don’t want to innovate for just your current customers.

6. Companies that product concept test can avoid costly consequences

In 1957, Ford spent the equivalent of $3.1 billion on the “car of the future,” the Ford Edsel. While Ford conducted consumer polls, they failed to implement feedback and many of the touted new features were unreliable.

Coors made a similar blunder when they attempted to enter the bottled water market in 1990. The packaging and branding of Coors Rocky Mountain Spring Water were similar to Coors beer, which frightened many of Coors’ target demographic.

Three examples of successful concept tests

Some brands have leveraged the testing process to make decisions that have tremendous payoffs. They can cover lots of areas, including:

• A product’s marketing strategy
• Design concepts
• Go-to-market strategy
• Positioning testing

A few examples include:

1. Emerging fashion brand Shinola used customer research to verify which watches to feature in their spring 2018 collection.
2. Yamaha used concept testing to make a product design decision about using a knob or a sliding fader in a new electronic keyboard. Understanding customer preferences has helped Yamaha maintain its position as the top keyboard brand in the industry.
3. Creative agency Linney Group uses concept testing with conjoint analysis to gauge its clients’ preferences and priorities, as well as what would resonate with their audiences. Data-backed strategies are agreed before going ahead with creative projects, saving time and money for the agency and its clients.

How do you conduct a concept test?

We’ve put together a step-by-step guide to help you create the best concept tests for your products and services. We’ll cover these topics below:

• Choose the right concept testing survey methodology
• Choose the right survey components for your concept test
• Choose the right flow for your concept test
• Identify the most promising concept

Concept testing common mistakes and how to rectify them

Or, you can also identify your most promising product concepts using our Concept Testing Program Solution. It’s an expert solution that tests and reveals what people think of your company’s concept ideas.

Step 1. Choose the right concept test survey methodology

For a successful concept test, in the early stages, consider which methodology will best fit your needs. Some of the most common survey methodologies include:

• Single Concept Evaluation (Monadic testing) – Respondents complete a full evaluation of a single concept.
• Multiple Concept Evaluation (Sequential Monadic testing) – Respondents complete full evaluations for multiple concepts. This method often includes a preference question to determine which concept is preferred, so it can be considered proto-monadic.
• Concept Optimization using conjoint analysis – respondents evaluate possible bundles of features to determine the most appealing.

Each product testing methodology comes with its own positive and negative trade-offs. Get a closer look at our pros and cons table.

Decide whether you want to optimize a single promising concept that you’re confident in, or if you have a range of possible features and price options that you’d like to combine in the optimal way. In the latter case, conjoint analysis may be a better option. In the former, single concept optimization may be a better choice.

Free eBook: Introduction to product concept testing

Step 2. Choose the right survey components for your concept test

Once you’ve determined your methodology, you need to decide what you’ll cover in your survey questionnaire. These will vary according to your purpose and the type of evaluation you’ve chosen.

Carefully consider the objectives of your concept testing to make sure the measures used will successfully answer your team’s questions.

As a general rule, you’ll want to measure the overall reaction to the concept in terms of acceptability, desirability and interest, plus the likelihood to purchase. A pricing component is also likely to be important for most concept tests.

Think about how the product might be used, and how it will fit into the respondents’ lives. Is it an innovative new category, or does it replace an existing solution? What situations might it be used in? How would the customer expect to buy it, and what would be their preference?

Consider the market segments of your target audience that you’re most likely to use, and order and prioritize them.

Step 3. Choose the right flow for your concept test

The flow of your concept test survey ensures that your target audience understands and buys into the survey they’re taking.

In the introduction section, make sure to introduce your survey in clear language.

Next: The main section. Here you can start to add in the core survey questions that introduce your concept and invite the respondent to evaluate its perceived value, relevance and appeal. Find out if the respondent sees the concept as a luxurious purchase or a functional necessity, whether its features seem well-thought-out, and whether it seems innovative. You may also want to establish a baseline of respondent’s buying history, asking about brands they have purchased in the recent past, quantities and usage frequencies.

End the survey by thanking the respondent. You could also use this opportunity to remind them that their data will be stored securely, and it’s only accessed by the research team.

Check out one example of a concept testing survey below. You can access this template for free

Step 4. Identify the most promising concept

When you carry out concept testing, you’ll receive a lot of data back. If you’re using a turn-key system like the Qualtrics Concept Testing Program, you can explore and filter it using visual dashboards. If you’re sorting the data by hand, use spreadsheet software like Microsoft Excel to begin to sort, filter, compare and pivot-chart your results. You can also try to pull information into a visual format with the in-built charting tools.

We recommend splitting your analysis into two sections:

• Overall results will give you a high-level view of which concepts performed the best.
• Individual results will let you dive into each concept and understand how they performed and why. Here you can apply your psychographic segmentation to uncover more from your results.

Where qualitative analysis for open-ended questions has been captured, try parsing out common themes, strengths, and weaknesses. Using a tool like Qualtrics Text iQ can make this analysis quicker.

Check out one example of a concept testing survey below. You can access this template for free, and begin using it immediately here.

Mistake #1: Running concept testing on one concept in isolation

You might choose to run a test to get your initial insights, but then you stop there. This might provide you with the answers you wanted, but you may be missing out on the full benefit of comparing research data over time.

Solution: In advance, think about what you’re measuring and how you’ll use that information when you get it. This might direct you to use a different methodology or suggest whether you need to repeat the exercise to see if your product’s development is on the right track.

Mistake #2: Testing too many concepts in one survey

You may want to use the tool to compare lots of concepts in one survey and ask a long list of questions for each, to save time. Even if you explain this in the introduction, it could make your target audience’s experience long, tiring, and confusing.

Solution: It’s important to respect the time and energy of the targeted respondent. To get an unbiased view of each concept, use the Monadic approach, which goes into depth on a single concept. For earlier stages of development, you can test multiple concepts in a single survey using the Sequential Monadic approach. Either way, keep in mind how many questions it’s reasonable to ask of your respondents.

Mistake #3: If you don’t get the answers you wanted in the concept test, you automatically cancel the idea

You find out that there is no demand or interest for your product ideas with your target audience. That could be a reason to stop investing your time and energy into the product’s development. However, this may be short-sighted as there may be potential profit in changing it.

Solution: The purpose of the research is to see what target customers think and feel about the idea. If it’s negative, then it could mean that the target audience is not interested at this time, or they are not the right audience. You may find more by testing more widely or trying again after some time, or consider adjusting the concept based on the feedback you’ve gathered. A bad result doesn’t mean it’s a bad idea. This is also where an early adopter question is useful — especially if you feel your new concept is ahead of its time or the market.

Mistake #4: You assume the audience thinks like you do

You’ve got an in-depth understanding of the product idea, and you want to share this in the survey introduction. You may also want to share marketing and brand ideas now, as you’ve been thinking of them already.

Solution: Avoid using technical jargon or acronyms that won’t be immediately understood by your audience. They can be intimidating and confusing for survey participants.

Also, avoid making a strong marketing pitch. Adding this information now can suggest the idea is set in stone – remember that marketing and brand activities come in at a later stage when the idea is close to or fully formed and you don’t want to prevent getting honest feedback.

Mistake #5: You don’t adjust the survey to cater to all geographies

You may have an audience you haven’t considered. Trial your test in different countries to avoid missing out on expanding your product idea’s reach.

Solution: Where you want to widen your audience pool, consider translating the survey into the native language. In addition, you can use images as survey answers, displaying them together to see which is preferred.

Learn more about concept testing

As we’ve seen, concept testing is more than getting feedback using a Likert scale. It involves considering the methodology, components, and flow of your survey for maximum impact.

It can be tricky to do and lots of companies are looking for support. Here are a few customers that turned to Qualtrics with their concept testing issues, and here’s how we helped them:

• Under Armour: This sports clothing and accessories brand struggled to scale its product testing with its current technology. By implementing the centralized Qualtrics Experience Management solution, they were able to increase product testing from 100, to 10,000 testers.
• Pinterest: This image-sharing and social media platform was trying to find out what would make their platform number 1 with their customers. Feedback on potential product developments was the starting point for an entirely new feature set.

When concept testing is done right, it saves a lot of time, money, and energy. Most importantly, this essential, money-saving tool gives you the opportunity to optimize the concept before development, based on the concept test feedback.