If you have attended any of the Alchemer Online Academy classes that I teach, you will surely recognize my propensity for answering questions with the phrase, “It Depends”.
This might make a funny joke that makes me and my co-presenter (and maybe even you?) laugh, but it does not help you with your questions.
For the rest of this article, I am going to present some of the most common questions that we get here at Alchemer, the simple blanket answer and the caveats to consider (the “It Depends” of each answer).
How Many Responses Should I Collect?
Simple Answer: 400
Caveats/It Depends: If you do not know the size of your population, but you do know that it is quite large, then you can choose 400 as the number of responses to collect. It is much better to use a sample size calculator to choose the exact number of responses to collect in order for your data to be statistically sound.
Population – A population is the entire group of individuals for which a conclusion is being drawn.
If your sample is not that large, you may not need 400 responses. Think about it. If your population is less than 400, than you will obviously need less then 400 responses.
If your population is small (about 50 or less), than you will need just about everyone to respond in order to get statistically sound data.
One more thing to consider is if you are going to segment your data. If you plan to segment your data for analysis, you will need to gather a statistically sound number of responses for each segment that you select.
Best to reference a sample size calculator or chart, like one found in this Alchemer article.
Should My Numeric Scale Have a Neutral Option?
Simple Answer: Yes, or no….
Caveats/It Depends: Well. I like to include a neutral option for my scale questions, except when I do not.
I tend to favor a 1 to 10 scale for business-to-business (B2B) projects and a 1 to 5 scale on business-to-consumer (B2C) projects. I like these scales just because this has worked best for me, over time. (Read: It depends on your project and target audience) I also like to keep my scales consistent across projects so that I can conduct trending analyses.
Trend Analysis – A Trend Analysis is an aggregation of data overt time and an analysis of changes that have occurred.
The question of which scale to use and if you should use a scale with a neutral option is an age old question, and we recommend knowing your target audience and to be clear on your survey goal and planned actions.
Neutral option fans favor this method because they feel like they are not forcing respondents to answer a question which they do not have enough information or experience with, or do not have any positive or negative feelings about.
Those in favor of scale questions with no neutral option, feel that they cannot take action on a neutral response. Sometimes inaction is the best action, but sometimes that is not a realistic option.
It is important to note that studies have shown that when lacking a neutral option, a respondent is more likely to respond positively than negatively. Now, this is interesting to know and keep in mind when doing your analysis.
If I see a majority of responses that are neutral, or close to neutral, I look to the open-ended questions to learn more. If I do not have these appropriately placed in my survey, I may do a side qualitative market research project in order to uncover the reasons behind the neutrality of responses.
You can find out more about scale questions here.
What is a Good NPS (Net Promoter Score) Score?
Simple Answer: Technically, anything greater than 0 is good and anything over 40 is great.
Caveats/It Depends: Now a good score really depends on your industry, yourself and how you define ‘good’!
To know if your NPS score is good or bad, you need to first define your competitive set. Will you be comparing your scores to others in your industry? Other departments throughout your company? Or your own team over time?
Net Promoter Score (NPS) – Is a management tool that can be used to gauge loyalty of customer relationships. It serves as an alternative to traditional customer satisfaction research.
If you are comparing yourself to your industry, well those numbers are very dependent on your industry specifics: airlines may have a much lower benchmark NPS scores than, say, online retailers. Know your competitors and then see how you compare. Note that industry NPS benchmarks are not always easy to acquire and they may have a cost associated with them.
If you are comparing internally to other departments, you will easily be able to acquire the competitive NPS benchmark, but you may have quite a different frequency and type of respondent depending on which individuals you are surveying. Those that are working with your sales team may have very different scores (and reasons for those scores) than those that are working with your service and support team. Capture this information, compare the scores, but do not punish a department if their scores are dependent on something out of their control.
And finally, if you are comparing your own scores over time, know that the best way to increase this score is to act on the feedback that you receive from respondents. Note that you may need to give yourself some time to implement those changes and see results in respondents.
Click here to learn more about the Net Promoter Score question type.
Why Hasn’t Everyone Responded to My Survey?
Simple Answer: Not everyone will!
Caveats/It Depends: You have worked really hard on your survey and after countless hours of reviews, re-writes and testing you are ready to send it out into the field. Why wouldn’t everyone feel the same way and respond instantly to your survey?
Well there are lots of things that can affect your response rate, and it really depends (there it is again) on whom you are sending it to, how you are distributing it and some details about your survey.
Response rate – Also known as completion rate or return rate, response rate refers to the number of people who answered the survey divided by the number of people in the sample. It is usually expressed in the form of a percentage.
First of all, let me say a thing or two about survey fatigue. Surveys are everywhere. Go to the store; survey on your receipt. Buy something online; you get a survey about it a few days later. Attend a festival; surveys are on all of the tables.
People just cannot get away from surveys. Now, I, as a survey-lover will take just about any survey that comes my way, but I am an anomaly. Most folks will only fill out a fraction of the surveys that they receive. Yours might arrive in the inbox of someone who is fatigued, on vacation, too busy or just not interested in providing feedback. So, response rates can vary.
Know your target audience, the type of person that you want to have responding to your survey. And know the best distribution method for them. If you are trying to reach web-savvy teens, an email campaign should work well. It you are trying to reach seniors in rural America, you may want to work a different method.
Once you have gotten your respondents to your survey, there are several factors that can detract them from completing your survey. When we see a large number of respondents drop off of a survey, we often look for some telltale signs that are causing fatigue or leading to survey abandonment. These include:
- The survey is too long
- Required questions without a viable option for all respondents
- Required open-ended questions
- Imagery and language that is not in-line with the survey brand
- Biased questions
- Questions require scrolling in order to be completed
Even if you have avoided all of the pitfalls of surveys, you can still expect your response rate to be much lower than everyone (100%). Industry standards state that surveys that are fielded to respondents that are familiar with your brand will receive an average of a 20 percent response rate; those that are not quite so familiar with your brand could yield as low as a 10 percent response rate. This is important information to consider when determining your sample size.
In Conclusion
Without knowing all of the details and idiosyncrasies of a project, it is almost impossible to answer a question accurately about a project.
Now, I know that this is hard to hear. Most people want a clear and simple answer to all of their questions, including their questions about market research. But as in any case, answering a question with a definitive, simple, blanket answer without knowing all the details can lead to problems and can also just be flat out wrong!
So, depending on your survey goals, learning objectives, target audience, length of survey and the details of your survey I can answer any of your market research questions, but for now all I can say is, it depends!
