Quantitative surveys have a common flaw. How do you know if the questions you’re asking represent the most important factors for your decision? If you don’t ask the right questions or overlook crucial factors, it can result in poor conclusions. Even worse, it might lead to no conclusion at all.
Let’s consider an example. Imagine you are building a product satisfaction survey. You want to figure out which product features contribute the most to consumer satisfaction. This way, your team can focus on developing more of those features. But your product has over four hundred features! How do you know which ones should be measured? Should you rely solely on the product manager’s opinion?
While the product manager may understand the product well, they may not fully understand the consumer’s needs. What if you both miss a key feature in the survey that could make or break the company’s future?
The solution is straightforward: conduct a qualitative survey sample before designing your quantitative survey. By asking a small group of respondents open-ended questions, you can uncover hidden factors that might otherwise be overlooked.
We call this qualitative exploration, but in plain English you can call it doing your homework on a survey project. By using this technique, you can uncover hidden factors that influence the topic of your survey. These factors might otherwise go unnoticed.
Qualitative Exploration
Incorporating qualitative research surveys during the exploration phase of your survey design offers two primary benefits. First, it helps you discover underlying factors that could be missed if you rely on intuition alone. Second, it enables you to build shorter, more focused quantitative surveys by eliminating irrelevant factors early on.
This approach is especially helpful when designing a survey for a product or problem you’re not familiar with. Qualitative research questionnaires allow you to gain a deeper understanding of the issues affecting your study. As a result, you can create a more accurate and actionable survey.
All you need is a survey for qualitative research, where you ask open-ended questions to a small sample of respondents. These questions are designed to elicit detailed responses, helping you uncover insights that will shape your quantitative survey.
Designing Qualitative Research Questionnaires
A well-designed qualitative research questionnaire should be short and simple, typically consisting of 2-3 open-ended questions. The goal is to describe a scenario and ask for the participant’s opinion or experience.
For example, instead of asking, “Please describe your experience,” try breaking it down into more focused questions like, “What parts of the experience did you enjoy?” and “Which parts did you not enjoy?”. This targeted approach will yield more specific and valuable responses.
Keep in mind that good survey design principles apply to qualitative and quantitative surveys alike. Avoid leading questions and refrain from suggesting responses. The goal is to receive honest, unbiased answers that can reveal factors you might not have considered.
Five Low-Cost Methods to Get Responses & Qualitative Data
When selecting good survey sample, the participants you select should be selected as randomly as possible to eliminate selection bias. When selecting a sample for an exploration project, it’s important to keep the same guidelines in mind. However, since you are not drawing concrete conclusions about the population from this part of the study, you don’t need to spend as much time on random selection.
Collecting the data from your study can be done in any number of ways, here are five of our favorites:
- Phone Interviews – One of the fastest and simplest ways to gather responses is through phone interviews. However, it’s important to reach out to participants in advance to avoid catching them off guard.
- Just remember that phones are a very personal possession, and many people don’t like being called or solicited for information out of the blue.
- We recommend emailing customers ahead of time asking them if they are willing to answer a few questions before you call them.
- Online Research Communities – Many organizations create online communities to engage participants in continuous qualitative research surveys. These communities can provide immediate access to an engaged focus group.
- They use low-cost social media tools like Facebook, Ning and SocialEngine to create the communities and maintain them with a small number of highly engaged participants that are offered incentives to answer their questions day and night.
- Once created, you basically have an on-demand focus group ready to answer your questions in a moment’s notice.
- The downside to this method is that maintaining a community requires time and continual effort — and it’s not much use if you haven’t set it up ahead of time for the project you need to do today.
- Use a Mini-Survey – Create a short, open-ended survey for qualitative or quantitative research purposes, and distribute it to a small group of respondents. Be careful not to overuse your survey population in the qualitative phase before moving to quantitative surveys.
- The caveat for this method is that you should not invite the respondents who take your qualitative exploration study to take your primary survey afterwards. Also, you only need a few quality respondents, so dole out the survey slowly so you don’t burn through your survey population on your exploration study alone.
- If you are using a panel company for your main survey – this is your best option for exploration.
- Use Existing Feedback Systems – Utilize existing customer feedback systems like support tickets or previous surveys. Be mindful of potential bias since those who contact customer support may not represent your entire customer base.
- Ask your service team for ticket content and feedback forms that have open text fields in them. You may also have past surveys and exploration studies you have done in the past. Reduce, Reuse and Recycle — it’s not just for the environment anymore!
- Of course, as with the other methods there is a “gotcha” here too –just because the data is available doesn’t mean it’s appropriate or unbiased.
- Customer support transcripts and tickets are a wealth of information, but the people that contact support may not be representative of your entire customer base. Remember to qualify the data before you base decisions on it.
- Ask the Front Lines – If direct access to customers isn’t feasible, consider interviewing customer service representatives who interact with your target audience regularly.
- It may not be as great a solution as going direct to the target population, but these folks usually have the best understanding of customer needs and issues in an organization. We recommend getting as close to the end customer as possible for these interviews. Avoid interviewing management becuase they tend to pre-filter their responses.
Incentives
If you’ve ever come to our Boulder trainings, or listened to my webinars, you’ll know that I’m not a huge fan of incentives for surveys.
I feel that surveys should be short enough not to need an incentive and the value of the survey should be so clear to the participant that they are more than willing to help without incentive.
That said, there is an exception to every rule. I do feel that participants in exploration studies and interviews often deserve (and need) something extra. One reason we avoid using open-text questions in our primary surveys is the time and cognitive fatigue they can cause. We are often asking a group of people to answer a survey that consists entirely of open-text essays, which can be overwhelming.
If you decide to offer an incentive to increase engagement just remember, the rules for good incentives still apply. Try not to bias your results with a bad incentive choice.
How Much Qualitative Data is Enough?
Determining the right amount of qualitative data can be tricky, as there’s no simple sample size calculator like with quantitative surveys. In general, aim for at least ten high-quality interviews for each segment of your survey population. This should be enough to identify key variables to inform your quantitative survey.
Analyzing Qualitative Data for Quantitative Survey Design
Once you’ve gathered qualitative data, it’s time to analyze it for patterns. Use your survey tool’s text analysis features, or export responses to Excel, to identify recurring themes. These themes will guide the creation of your quantitative survey questions and response options.
However, be cautious of allowing your opinions to shape the results. It’s easy to misinterpret open-ended responses, which is why it’s crucial to follow up with a quantitative survey to confirm your findings and provide statistical validation.
Bringing it Back to the Survey
Once you’ve collected the data from your survey/phone calls/interviews you need to analyze it to look for patterns. This is where qualitative exploration and qualitative data analysis comes in.
Use your survey tool’s text bucketing feature or export all your responses to excel and start looking for recurring patterns that identify variables that effect your learning objective. Take those variables and use them when creating your survey’s questions and multiple choice options for your survey. You’ll likely also start seeing overt trends in the data at this point — but don’t act prematurely!
The greatest threat to the process at this point is your opinions and intuition. When analyzing open ended responses there is always a chance that the researcher may misinterpret or represent what they survey taker intended to say.
Qualitative data is insightful and useful — but it’s not a statistical representation of your population’s opinions and demographics.
That’s why you do a quantitative survey based on this discovery data. To provide statistical certainty to the directional and qualitative data you’ve collected.
Discussion
* Do you use qualitative research when designing surveys? What methods have worked for you?
* Do you have questions about quantitative vs. qualitative?
* Try it out: brainstorm use cases in your organization for qualitative exploration.
