Generating Test Responses with Q_PopulateResponse | XM Community

Generating Test Responses with Q_PopulateResponse

  • 12 March 2018
  • 6 replies
  • 471 views

Userlevel 7
Badge +27
  • Level 8 ●●●●●●●●
  • 5486 replies
A conversation I had with another community member at the X4 Summit reminded me that others might benefit from the process we use to generate test responses. This is a long post, but I've tried to keep it as focused and concise as possible. My goal is to help you make effective use of the Generate Test Responses feature in Qualtrics.

Why Generate Test Data?
Generating a robust set of test data and carefully examining it ensures that your survey delivers the data you expect and helps avoid costly mistakes. By covering every possible answer option and logic branch, it can be used to uncover logic errors that may have been missed in manual testing or flaws in the survey design.

Issues with Generated Test Responses
The primary problem with Generate Test Responses that many run into is that it can be nearly impossible get past screening questions randomly. You may need to generate thousands of responses to get one complete. Some people, myself included, have added variables and survey flow logic to force test responses past screening questions. The problems with that approach are that it can be complex to set up, partially invalidates the test, adds the risk of introducing new errors, and makes the resulting data confusing.

Q_PopulateResponse to the Rescue
Q_PopulateResponse is a parameter you can add to your survey url to pre-answer questions. When used with Generate Test Responses, the questions specified in Q_PopulateResponse are answered as specified instead of randomly. Using Q_PopulateResponse with Generate Test Responses allows you to create a robust set of test data with relatively few test responses and no testing specific variables or survey flow logic.

A Q_PopulateResponse parameter value specifies the questions and their answers as a json string. Here are some examples (The json string starts with { and ends with }. Also note the answer values are choice ids, NOT recodes):
1. Multiple Choice, Single Select: {"QID1":"1","QID2":"3"} //QID1 answer 1, QID2 answer 3
2. Multiple Choice, Multiple Select: {"QID1":"1,3"} //QID1 answers 1 and 3
3. Matrix, Single Select: {"QID1":{"1":"2","2":"3"}} //QID1 choice (row) 1 answer (col) 2 and choice 2 answer 3
4. Multiple Choice, Carryover: {"QID2":"x1"} //QID2 carryover choice 1
5. Text Entry: {"QID1":"text"} //QID1 text value

Creating the Q_PopulateResponse Json String
Here is what I've found to be the easiest way to generate the json string:
1. Manually take the survey to completion
2. Download a CSV of the data and open in Excel
3. Transpose the data to a new sheet
4. Build Q_PopulateResponse strings for each screening question and other questions where you need to force an answer using concatenation formulas. To include a quote in your string you need two quotes. So, a formula to generate the string "QID1":"1" where QID1 is in cell A1 and 1 is in cell B1 would be `=""""&A1&""":"""&B1&""""`.
5. Concatenate the strings for all the questions into one comma separated Q_PopulateResponse string. In newer versions of Excel (e.g. Office 365) use the TEXTJOIN function to concatenate strings `="Q_PopulateResponse={"&TEXTJOIN(",",TRUE,D1:D20)&"}"`. If using an older version of Excel, adding a vb concatenate range function to Excel makes this easy: `="Q_PopulateResponse={"&concat(D1:D20)&"}"`. Here is a link to the concat function code: https://gist.github.com/marketinview/141bb2307fac90a6c94c

Generating Test Responses
Using the following process will ensure that responses are generated for all possible scenarios of completes and screen-outs.
1. Check Show Verbose and generate one test response with the complete Q_PopulateResponse string to make sure your string results in a complete. If needed, modify and repeat until you have the string correct.
2. Generate more responses with the complete Q_PopulateResponse string. The number of responses needed will depend on the complexity of your survey. Generally, you don’t need that many - usually from 25 for a simple survey to 100 for a complex survey.
3. Remove the last question from the Q_PopulateResponse string and generate more test responses. The number of responses needed depends on the number of choices/answers in the question you removed. A rule of thumb is 3x the number of choices/answers.
4. Repeat the process in (3) until the Q_PopulateResponse string is empty, then remove Q_PopulateResponse completely and generate test responses one last time.

If you've read this far, I hope I've provided enough information to get you started. Feel free to post follow-up questions.

6 replies

Userlevel 7
Badge +7
This is one of the best things I have seen in quite a while, and will help with getting quality test data a TON. Thanks @TomG
Userlevel 7
Badge +27
Thanks @AnthonyR!
This is great. Is there a way to populate a multiple-choice text entry with a numeric value? I have tried altering your code but I can't seem to get it to work.

Tom -- I finally got around to testing out this mechanism, and it is indeed very helpful but I am having some issues... I've verified that I can string together screener variables and get a set of responses that all pass the screener. I did run into challenge for a while related to "recoded" values -- Qualtrics is a little annoying in that if you enable recoding values, you can't toggle it off to see what the original values are -- you have to write them out to the CSV. But I figured that out after head scratching.
I have a few questions about the capabilities here:
Is it possible to set a question to not be a certain value (i.e. if I have a question with 5 choices and a "None of the above" could I specify that it can be any of the 5 choices except None?).
Also is it possible to set it so that a single-select question could be either of a few values (e.g. the equivalent of "QID2":"1" or "QID2":"2"). This is sort of a different way of getting at the previous question. Or do I need to run test responses for each of those single-select options? (that's what I'm guessing given the JSON structure).
Thanks!
Bob

Userlevel 7
Badge +27

Is it possible to set a question to not be a certain value (i.e. if I have a question with 5 choices and a "None of the above" could I specify that it can be any of the 5 choices except None?).

No, it isn't possible.
Also is it possible to set it so that a single-select question could be either of a few values (e.g. the equivalent of "QID2":"1" or "QID2":"2"). This is sort of a different way of getting at the previous question. Or do I need to run test responses for each of those single-select options?

Yes, you have to generate test responses for each of the single-select options.

Userlevel 1
Badge +1

https://community.qualtrics.com/XMcommunity/discussion/comment/18271#Comment_18271Hi Dman, Did you get this working for multiple choice? Having tried the solution above to no success. have posted issues identified on this new post
https://community.qualtrics.com/XMcommunity/discussion/22392/how-do-you-get-the-embedded-data-from-coded-questions-to-populate-a-follow-up-survey#latest

Leave a Reply