About A/B Testing in Website Feedback
A/B Testing is a classic research format used across disciplines. In this testing method, you compare a control condition to one where a treatment is applied, so you can then analyze if there is a significant difference between the respondents’ behavior in the treatment vs. the control baseline. A/B Testing can be useful in web development because it allows you to compare two versions of a design or advertisement in order to determine which is preferred by your customers.
On this page, we’ll explain how to test two different versions of the same Website / App Feedback creative to randomized subsets of site visitors, to see which performs better.
Establishing the Creatives Being Tested
Creatives are the visual element of a Website / App Feedback project. The first step in A/B testing creatives is, of course, to make the creatives you want to test.
Whether you are trying out different ads, banners, or embedded content, you can A/B test just about any creative. To help you get started, here’s a list of all the creatives you can make in Qualtrics:
- Responsive Dialog
- Feedback Button
- Embedded Feedback
- Pop Over
- Info Bar
- Embedded Link
- Pop Under
- Custom HTML
Once you’re finished, make sure you publish your creatives so that they are pushed to the live code. Only changes that are published will display on your website once you have the code placed on your website.
If you are editing a creative, have the code deployed to your website, and do not want to push your edits live to website visitors, click Save instead.
Setting Up Intercepts
Once the creatives are made, it’s time to move onto the intercepts. Intercepts are the rules governing when and how your creatives are displayed on your website.
Every intercept is made up of at least one action set. An action set determines what happens on your website when certain conditions are met. When you set up multiple action sets, you can start to do more complex things, like present a completely different creative to different sets of website visitors.
- Create an intercept.
- Create an action set for each creative you want to test.
- Add logic, as needed. For logic that should apply to both creatives, use intercept targeting logic. If it’s creative-specific, use action set logic.
Example: Regardless of which randomized subset sees my banner, I only want it to appear on my support pages. I specify that the URL has to contain “/support/” in the intercept logic.
- Go to your intercept Options and select Randomize action set.
- Make sure your Display sample rate is 100%.
This step is optional, but can change how dynamically you can report on the results of your A/B test. If you want to pass certain information over to your website, use CX Dashboards, or set up advanced reports in a target survey, you’ll want to add embedded data to your intercept.
Embedded data are custom variables that can be named and set equal to any values you desire. For the purpose of A/B testing, that means you can track information related to which treatment / version of a creative your website visitor was exposed to.
- On your first action set, select Options.
- Select Embedded Data.
- Name your field – something simple like “Condition” works.
- Choose Static Value.
- Give this condition a unique label.
Qtip: You can just number your conditions, or you can choose more descriptive names. Whatever you think will benefit you most when reporting this data.
- Save your changes.
- On your second action set, select Options.
- Select Embedded Data.
- Use the exact same field name as Step 3 – for example, “Condition.”
- Choose Static Value.
- Give this condition a unique label. It must be different from what you wrote in Step 5.
Example: If your other action set is “Version 1,” this action set could be “Version 2.”
- Save your changes.
- Repeat as needed if you have more action sets.
Once your different conditions are marked, you can create additional embedded data that will help you build robust reports on the performance of your creatives. For example, you can capture how long the visitor spent on the site, how many pages they visited, the URL they were on when they saw the creative, and so, so much more. Check out the Embedded Data Value Types section of the Embedded Data in Website Feedback for your options.
Qtip: Remember that for any additional embedded data you capture, you need to make sure it’s added to both action sets, or you will only have that data for one of your conditions. Here’s what the same embedded data shown above would look like for your second condition:
Qtip: Once you choose targets for your action sets, don’t forget to add the appropriate query strings. Generally, you’ll be adding the same target to each creative, but you’ll want to make sure values in your query string differ where necessary.
Additional Intercept Customization
The rest of the options and customization you add to your intercept will depend on the needs of your program.
When editing an intercept, think about how you want that specific touchpoint to behave towards website visitors. For more guidance on the options available to you, see:
- Setting Up Your Intercept: An introductory page that goes over top-level information on intercepts and the types of edits you might want to make.
- Editing Intercepts: Step-by-steps on all the basics to intercept editing, from targets, to the rules under which they display (logic), and more.
Additional resources and advanced options:
- Multiple Action Sets: We covered this a bit in the steps above, but this page goes over how to create and make the most of multiple action sets in your intercept.
- Embedded Data in Website Feedback: We covered this a bit in the steps above, but this page goes more into depth over how embedded data can be used in Website / App Feedback projects.
Qtip: Be careful when editing the pre-configured embedded data. If you remove or rename fields, you will not capture the desired information.
- Intercept Options: This page goes over additional settings that can enhance the behavior of your intercept. Many of these settings manage how frequently visitors on your site see the intercept.
Deployment to a Website
Deployment for A/B testing is the same as for any other Website Feedback project.
Testing and Activating Intercepts
Try testing the intercept on your website before you deploy anything.
When you are finished editing your intercepts, publish them.
When you are ready for your edits to be live on your website, activate your intercepts.
Getting Deployment Code for your Developers
Resources for Development Teams
We have a few resources that can help your development team, once they have the code:
Each creative you made will come with a Statistics page. Here, you can get an idea of how many clicks and impressions your creative generated over a certain time period, and even compare two creatives against each other. You can even download this data to a spreadsheet.
If you added embedded data to your intercepts, then this opens up a world of possibilities with regards to reporting.
Embedded data collected through a Qualtrics survey can be reported on just the same as any other survey data. This means it can be integrated into survey reports and CX Dashboards, and tell a larger story with all the data collected in your program.
Here are some pages to get you started:
Example: We were testing two different feedback intercepts. These number chart widgets shown side by side compare how many responses each version collected.
Example: A simple chart widget displaying comparison of time spent on the site after exposure to one version of the creative vs. another.
Example: A bar chart comparing responses for each version of the creative.