For How Long Should Your Survey Be Open?
One of the most common questions you will be asked about your survey is “What was the sample size?”, and rightly so. Sample size is an important signal that tells others if they can believe your findings. Sample size is what makes the difference between your research being “quantitative” that is generally statistically significant or “qualitative” that is generally not statistically significant.
How long your survey is active will depend on your sample size and recruiting method (with a panel? Email? Social media? Word of mouth?), but here are some general tips to help you determine how long to leave your study in field.
Knowing when to close your survey can be difficult to determine, but a good guideline is to run a check on three markers daily to take stock of the progress you are making against your sample quota. Make a simple graph of your daily response rate by tracking:
- Good Completes: How many respondents successfully finish your survey
- Screen Outs: How many respondents did not qualify to take your survey
- Abandons: How many respondents qualified for your survey but abandoned before finishing
After a week or less of tracking this data you should be able determine if your current recruiting efforts will be enough to get you across the response finish line (your quota) and by when.
Do a Soft Launch
You should always run a soft launch of your survey before completely opening up the spigot in full. Generally a soft launch is 10% of your total recruiting list if using email. A soft launch is a sneak peak into what you can expect for response rates, screen outs and abandons. It allows you to adjust anything wrong with the flow of your survey, the length of your survey and your screening criteria so those friction points don’t exist when you full launch. Soft launches are also a great way to catch any errors in your survey flow or logic.
Make Peace with Response Rates and Timeframes
It’s understandable. You work hard to design a survey and you hope others are excited to take it. Sadly, this isn’t usually the case. Most people will ignore your survey just like most fish will ignore a lure. Part of accepting this fact means understanding the normal time frames it takes for respondents to respond.
For an email invite, you can expect to see almost half of all your respondent activity within the first 3 days of invitation. For social media invitations the bulk of your responses will come within the first 24 hours of posting. Sure, you will see responses flow in after but they will come in more slowly until they are just a trickle.
Because of this law of diminishing returns, it doesn’t help much to leave your survey in the field for extended periods of time just hoping something ignites a fresh round of responses. If you haven’t made a serious dent in your quota by day 3 or 4, you probably need to change tactics, increase your incentive or take a new approach to recruiting.
While there is no hard-and-fast rule for knowing how long to keep your research live, these tips should give you the right clues to indicate if you will reach your target sample size and by when.
March 26, 2020
Social distancing in market research: Pivoting in-person methodologies
March 20, 2020
4 things market researchers need to think about in the time of COVID-19
February 26, 2020
Calling all brand admins: Join the Qualtrics admin training at XM Basecamp Live
February 14, 2020
Major League Baseball’s Scott Harris on surprising insights and the future of research
February 13, 2020