Survey Platform - Screen-out Management | Qualtrics

Screen-Out Management

Introduction

What happens when a respondent doesn’t meet your qualifications for completing the survey? By default, all responses that are submitted, regardless of which questions are answered, are saved and considered “Complete”. All collected responses will count toward any response limit on your license.

In addition to the default behavior, there are three alternative methods to handle screened out respondents:

  • Save all responses, but flag those that are screened out with Embedded Data for easier identification and filtering in the results.
  • Discard screened out responses so they are not saved and do not count toward any response limit on your license.
  • Discard screened out responses, but keep a tally of how many respondents screen out at different points in the survey (also known as “Incidence Reporting”, this method is an add-on feature not included in the standard Qualtrics license).

This page further details the default behavior for screened out responses and outlines these methods of managing screen outs in the Survey Flow.

Recording Screened Out Responses

By default, Qualtrics records any respondent who makes it to an ending point in the survey as a complete response. This includes respondents who answer all questions, as well as those who answer a few questions and then are sent to the end of the survey with Skip Logic, Display Logic, or Branch Logic. All these responses will show in the results and in the downloaded dataset as “Complete” responses.

image23
“Completed” here refers to any respondent who made it to an ending point in the survey.

image00
In the Finished column of the downloaded survey data, “1” refers to any respondent who made it to an ending point in the survey. Columns F, G, and H are hidden to make the screenshot more readable.

Screening Out Respondents in the Survey Flow

Methods for managing screened out responses—such as flagging, deleting, and tallying screen outs—are all handled in the Survey Flow. To use any of these methods, you’ll first need to screen respondents out using the Survey Flow. This process consists of two steps:

  1. Skipping respondents ahead to the end of the question block they are currently answering
  2. Using a Branch in the Survey Flow to end the survey for those respondents.

To skip respondents to the end of a question block

Qtip: Skipping screened out respondents to the end of the question block makes it easier to end the survey immediately when a respondent no longer qualifies for the survey. It also saves the respondent the time they would have spent answering the rest of the questions in that block.
  1. Navigate to the Survey module and select the question that will screen out respondents.image12
  2. Open the Question Options menu and choose Add Skip Logic.image21
  3. Set up logic in the first two dropdown menus to reflect those who should be screened out from the survey.image29
  4. Skip those who meet the criteria to End of Block and click Done.image18
Qtip: If you don’t need to delete, flag, or tally screened out responses, but instead would like them recorded in the dataset just like any other response, you can choose to skip respondents to End of Survey instead of End of Block and ignore all of the steps below.

Branching Respondents Out of the Survey

Now that screened out respondents have been skipped past the questions they no longer qualify for, we can use the Survey Flow to screen them out of the survey.

  1. Navigate to your Survey Flow in the Survey module.image15
  2. Click Add Below on the question block that contains the screening questions.image02
  3. Choose Branch and set up logic to capture those who should be screened out of the survey.image20
  4. Click Add a New Element Here underneath the Branch.image04
  5. Choose End of Survey.image09

With the steps above, respondents are now screened out in the Survey Flow rather than with just Skip Logic. This method opens up the options to flag, delete, or tally these responses. See the following sections for more information on how to implement these options.

image03
With this Survey Flow, all respondents see the “Introduction and Screener Questions” question block. Those who are under age are sent to an early survey ending. Everyone else continues to the “Scored Section” block.

Flagging Screened Out Responses

One common scenario for handling screened out responses is to save them in the dataset and flag them with Embedded Data for easy identification and quick data filtering.

Embedded Data is any information you choose to save with survey responses in addition to the answers to the survey questions. In this case, you might save an Embedded Data field with the name “Response Type” and set it equal to “Partial”, “Screened Out”, or “Complete”, depending on where the respondent exits the survey.

To flag screened out responses

  1. Screen out respondents using the Survey Flow, as described above.image03
  2. Click Add a New Element Here below the End of Survey element.image26
  3. Choose Embedded Data.image05
  4. Click anywhere on the Embedded Data element and press the key to move it directly above the End of Survey element.image07
  5. Type a name for the Embedded Data field and click Set a Value Now.image08
  6. Type a value for the Embedded Data field.image30

In the results, Embedded Data is reported along with the survey questions. In the example above, a field called “Response Type” will be included in the data, with entries of “Screened Out” for respondents who completed the survey this way.

Qtip: If you just want to flag screen outs and you don’t need to mark partials and completes (or any other fields), you can use the Flag Response As option in the End of Survey element and set it to Screened-Out.

Embedded Data can also be placed at other points in the Survey Flow to track other places where respondents leave the survey.

Example of flagging responses

Let’s say we want to assign values of Partial, Complete, and Screened Out to our respondents. Our Survey Flow might look something like this:

image13

All respondents at the start of the survey are assigned a “Response Type” of “Partial”. Those who are screened out after the screener questions have “Response Type” changed from “Partial” to “Screened Out”. Those who either finish the entire survey have “Response Type” changed from “Partial” to “Complete”. Those who do not finish the survey will remain with a “Response Type” of “Partial”.

With the above Survey Flow, a table such as the following visualization could be included in your Reports, outlining how many respondents fall into each of the three categories.

image32

Discarding Screened Out Responses

In some situations, you may not want to save the responses of those who are screened out of the survey. Discarding these responses ensures they do not count toward any response limits on your license.

Warning: Responses that are not saved cannot be retrieved later. Use this method with caution.

To discard screened out responses

  1. Screen out respondents using the Survey Flow, as described above.image03
  2. Click Customize on the End of Survey element.image11
  3. Enable Override Survey Options.

    image01
  4. Select Do NOT record survey response.image31
  5. Click OK and save the flow.image22

Tallying Screened Out Responses

Tallying represents the best of both worlds—screened out responses are not saved in the results, helping to limit the total response count needed for each project, and a tally is provided that identifies how many respondents screened out at each point.

This option, known as Incidence Reporting, is not included in the standard Qualtrics license. You can check if it is available for your account by following the steps below. If you have access to the “Screen-out Response” option in step 3, your account has Incidence Reporting enabled.

Qtip: If you do not have Screen-out Response as an option, please contact your account representative.

To tally screened out responses

  1. Screen out respondents using the Survey Flow, as described above.image03
  2. Click Customize on the End of Survey element.image11
  3. Enable Override Survey Options.

    image01
  4. Select Screen-out Response and type a name for the screen-out. This name will help you analyze the screened out responses.image10
  5. Click OK and save the flow.image27

Answers for respondents finishing the survey at this point will not be saved, but a tally of these respondents will be kept.

The steps above show the creation of one general screen-out point to capture all respondents who exit the survey. To tally why each respondent screened out, follow the steps in Screening Out Respondents in the Survey Flow and the steps above, but create a separate Branch for each screen out condition, each with a different screen out name.

To view your screen-out counts

  1. Open your Survey Flow.image15
  2. Click Customize on any End of Survey element.image11
  3. Click Show Screen-out Counts.image24
  4. View your tallies for each screen-out category.image16
  5. Click Close when finished.2015-07-17_16-02-59

Using Screen Outs with the Qualtrics Mailer

If you have Screen-out Response enabled and you send your survey through the Qualtrics Mailer, you will be able to identify which contacts were screened out.

  1. Navigate to the Emails tab in the Distributions module.image14
  2. Click the dropdown arrow to the right of your email distribution.image17
  3. Choose Download History.image19
  4. Check the Status column in the CSV file for “Screened-out” or any of your custom screen-out names.image28

The Distribution History provides a list of all contacts you emailed the survey to, as well as a number of other details. The Status column indicates who was screened out and gives the Screen-out point name.