Variable Rules CHANGED retroactively with no solution for fix

Kathryn_AnnKathryn_Ann San Francisco, CACommunity Member Qubie ✭

Variable Rules CHANGED retroactively with no solution for fix

At some point in the past few months, Qualtrics engineering changed how variables worked. They didn't both to even tell those in their support department, let alone their customers. So, you all may be aware, here is the info. If a question used in a variable was skipped or opted out, the variable now adds in a 0. Prior to this change, a blank was left and no calculation would be made. For example, if you wanted to get a simple mean across 5 questions you would use a variable Mean=(a+b+c+d+e)/5, if a survey contained answers to all five questions the variable would be populated with the mean. If one or more of the questions was opted out or not answered the variable would be left blank. Thus, only true, valid means were calculated. NOW, in either case a 0 is added in, which completely distorts a rating to the negative. An opt out or no answer should not count as a 0, but it does! I have MANY such variables in MANY surveys, which are all now useless. The "solution" is to setup Scoring which gets piped into an embedded data field. However, it will calculate a weighted mean for ALL respondents, even if they opt out of 4 of the 5 questions. Thus, there is no way to make rating for only those who answered all questions in a set. In addition, it only works for future NEW survey data as the score data is not available in data and analysis nor in reporting. Therefore, this makes all historical data not useable, unless you export and calculate outside in excel. But, why setup scoring at all if you are going to export and calculate in excel??

Tagged:

Comments

  • uhrxx005uhrxx005 Community Member Guru ✭✭

    Hi @Kathryn_Ann, thank you for making me aware of this. I think I may have found a solution to the issue. If you go into the scoring section and click on "scoring options" on the top right there is a checkbox that says "treat empty statistics for scoring categories as 0". Unchecking that box may fix the problem going forward. I don't know if that will fix the issue retroactively.

  • Kathryn_AnnKathryn_Ann San Francisco, CACommunity Member Qubie ✭

    I tried this with a support person and it didn't solve the issue. :-(

  • EmilyEmily Seattle, WACommunity Administrator Administrator

    Hi everyone! Just wanted to jump in here and provide a bit of clarity. The option that @uhrxx005 suggested will work if set up initially before data is collected, but unfortunately will not apply retroactively.

    However, our Support Team has put in a feature request for this to be possible. Therefore, if you would also like this functionality, I highly recommend reaching out to our support team and letting them know so that they can also indicate your interest in the feature request!

  • uhrxx005uhrxx005 Community Member Guru ✭✭
    edited May 2018

    @Emily, Why would Qualtrics think its okay to change something as significant as this without having a fix that works retroactively? Either Qualtrics knew this would break survey calculations and did it anyways, or they didn't do their research. Fortunately, I don't have any active surveys with variable calculations active at the moment, but if I had when Qualtrics made this change I would be very frustrated. This has likely broke a ton of active surveys and I would think this should be of the highest priority for Qualtrics to fix. It shouldn't require a dozen of us contacting them and requesting a fix.

  • Kathryn_AnnKathryn_Ann San Francisco, CACommunity Member Qubie ✭

    @uhrxx005 Thank you for your Support and many others may not realize that their calculations are now all wrong, as you have to actually look at the math being done to realize that it is incorrect.

    @Emily- actually, the “fixes” that I did with support are not working even for new data

  • EmilyEmily Seattle, WACommunity Administrator Administrator

    We do value your feedback, @Kathryn_Ann and @uhrxx005, and can understand how frustrating this can be. We have personally passed along and expressed your concerns to our product engineering team so that they can take your situation into account. We apologize for the inconveniences that this has caused you during your data collection. For anyone else who would like to investigate if their survey data has been affected, this change went into effect on February 1st.

Sign In to Comment