Designing a survey or questionnaire involves much more than just choosing what questions to ask.
There are multiple factors that influence your project’s success or failure, so be sure to consider these 10 key factors when you’re creating your survey.
Keeping these in mind will help ensure that you collect high quality data that you can act on with confidence.
1. Your Mode of Data Collection
When you’re gathering data through phone surveys or in-person interviews, the interviewer’s words and actions will have a major impact on your final results. Online surveys, on the other hand, rely heavily on question wording and design.
You won’t have the benefit of inflection or body language to help convey your questions’ meaning. Phrasing, color choice, and layout will all play major roles in how respondents interpret your questions. Their interpretations will influence your final data.
2. Impact of Survey Fatigue
There are question types, such as huge grids of radio buttons, that place a heavy burden on your respondents and increase their mental fatigue during the survey.
Sometimes these questions are necessary, but avoid them if you can.
Similarly, you don’t want to write convoluted questions that tax respondents’ working memory or comprehension. These poorly designed questions tend to elicit higher non-response rates and may impact your ability to act on your data.
3. The Effect of Survey Question Wording
Writing questions with the right wording is one of the most difficult parts of creating surveys. Asking a question doesn’t sound that complicated, but remember that collecting data online removes the ability to explain your meaning or offer additional information.
If you use esoteric language, inaccurate terminology, or highly technical words, your respondents may not be willing (or able) to provide answers.
It’s also vital to avoid asking two questions at once (e.g. “Please rate your experience with the food AND drink”) or writing questions that bias respondents toward a particular response.
Providing clear instructions without nudging people toward one answer or another is a delicate balancing act.
4. How You Order Your Questions
Each survey question should follow a logical flow. Jumping around from topic to topic may confuse your respondents and cause them to skip questions or abandon the survey altogether.
Be sure to carefully test your survey once it’s completed so you can identify any inconsistencies in the survey taking experience. If you’re using page or question logic to change what questions individuals may see, you need to take the survey multiple times so you follow all possible paths to completion.
5. Different Survey Question Formats
Not all survey questions are created equal. Some, known as quantitative questions, will provide a limited selection of available responses. Qualitative questions, on the other hand, allow people to answer in their own words.
Each type of question serves different survey goals and has its own limitations.
The key is to understand what level of detail you need to achieve the aims of your project so that you can incorporate the right blend of quantitative and qualitative questions.
6. Accuracy of the Answers You Receive
Most people don’t have a flawless memory. Regardless of their intentions, respondents will not always be able to provide you with accurate information.
For instance, people can answer questions about their gender and age easily, but when it comes to measuring attitudes and opinions, many people have trouble formulating an answer.
To get the best possible data about these subjective topics, use language the reflects how your respondents actually think and talk about the topic you’re asking about.
7. Bias in Self-Reported Behavior
Generally speaking, people have less precise memories of the mundane behaviors they engage in on a regular basis, and they usually do not mentally categorize events by periods of time (e.g. week, month, year).
We should consider appropriate reference periods for the type of behavior we want them to recall. For example, respondents can probably tell you how much time they’ve spent commuting to work this week, but if you ask them how long they commute in a year their recall won’t be as reliable.
Ultimately, questions about measured behavior should be relevant to your respondents and capture their potential state of mind.
8. Clear Question Structure
Survey questions have three distinct parts, and each must work in harmony with the others to capture high quality data.
These three parts are the question stem (e.g. What is your age?), additional instructions (e.g. Select one answer), and response options (e.g. Under 18, 19-24, 25+). The wrong combination can leave respondents confused about how to answer a question.
Confused respondents lead to confusing survey results.
9. Visual Survey Design
The verbal part of your survey is crucial, but if you’re using survey design elements in an inconsistent way can increase the burden on your respondents and prevent them from fully understanding your survey’s meaning.
For example, using different font sizes, colors, and strengths across questions forces the respondent to relearn their meaning each time they’re used. Also, presenting scale questions with different directions (positive to negative on one question, then negative to positive on the next) within the same survey dramatically increases measurement error.
Respondents may assume that all rating questions have the same scale direction even if individual question instructions explain the meaning of the scale’s end points.
10. Your Final Survey Analysis Plan
Both the type of information you request and the question format that you choose should be based on your research objective and the type of analysis you plan to perform once the data is collected.
You’ll also want to consider whether you want to replicate the survey results, track certain events, or just run a one time ad-hoc analysis on your results. If your goal is to track certain metrics, for example, you should carefully craft tracking questions, as slight changes in wording can totally change the meaning of a question (and thus change its results).
If you take each of these aspects of survey design into consideration, you’ll be well on your way to creating surveys that produce valid data that will allow you to make tactical and strategic business decisions with confidence.
