Isn't "randomly present elements" unnecessary for high enough numbers of participants? | XM Community
Solved

Isn't "randomly present elements" unnecessary for high enough numbers of participants?

  • 14 October 2020
  • 7 replies
  • 682 views
C
Badge

I am having trouble understanding the "evenly present elements" option.
I am designing a survey with 3 conditions and I'd like to randomize the participants into those conditions. Currently I have set up a randomizer with 3 embedded data elements which assigns one of three values (1,2 or 3) to the variable "CONDITION". I set the number of elements presented to a participant to 1.

So far so good. But should I check the "evenly present elements" option? If what condition one is assigned to is randomized, i.e. there is a 33.3% chance in my case, then in the end there will be roughly the same amount of participants in each condition, without having to execute any feature that "evenly presents" the options, right? If I throw a coin 100 times, it's gonna be about 50 times heads and 50 times tails.

So what is this feature good for?

icon

Best answer by TomG 15 October 2020, 20:30

View original

7 replies

P

On average, yes. But it might come up heads 98 times. This feature helps guard against the unlikely, which becomes less important as the number of responses goes up.

C
Badge

Is using this feature still appropriate for randomizing participants in a controlled experiment though? I don't quite understand it, and I'm unsure whether I should use it or not.

TomG
Rank
Userlevel 7
Badge +27

https://www.qualtrics.com/community/discussion/comment/31170#Comment_31170Yes, you should use it. It will ensure that respondents are randomly assigned to your three conditions evenly instead of just randomly. It will reduce the variability in how many respondents are assigned to each condition. It reduces the need to over sample and makes statistical analysis better since you are less likely to run into issues of small sample sizes if you cut the data by other criteria.
It does not guarantee that you will have equal numbers of completes because people could abandon the survey after being assigned but before completing, but it improves the odds.

C
Badge

https://www.qualtrics.com/community/discussion/comment/31171#Comment_31171Great, thank you for your insights. I will use it then.

P

https://www.qualtrics.com/community/discussion/comment/31171#Comment_31171Thank you!
I'm still not fully convinced about what it means for the scientific rigor of an experiment.
Is there also a more profound discussion of this somewhere? E.g. how does the evening out work in qualtrics? What assumptions do I have to make to argue this is not causing any biases?

A
Rank
Userlevel 7
Badge +21

I doubt you'll find a more detailed explanation of these anywhere. Qualtrics isn't aimed at scientific studies, its more for business purposes.
From my small testing of the randomizer (3-4 experiments, 10-12 draws, 2-4 elements in the randomizer), here's the difference that I noticed.

  • Without evenly present: Qualtrics does a random draw and presents one element to the respondent.

  • With evenly present:

  • Qualtrics checks the number of times each element has been presented, shows the one which has been presented least

  • If more than one element has been presented the same number of times (lowest), it does a random draw to present one

  • It does this till all elements have been presented an equal number of times


But as for the main question, does it have an impact for a large enough sample, I don't think it should make a difference.

TomG
Rank
Userlevel 7
Badge +27

https://www.qualtrics.com/community/discussion/comment/34518#Comment_34518It works by keeping count of how many times it picks each item in the randomization. I can't speak specifically to the Qualtrics algorithm, but typically it is done by first shuffling the set of items, then sorting them by count. Once that's done, as many items as are needed are picked from the top of the list.

Leave a Reply


Terms & Conditions