Bret is our resident expert on surveys and collecting data with panel responses here at Alchemer. He’s spent hours helping people get the most from paid panels, and I caught up with him this week to ask a few questions about survey projects and getting the best survey data with panels.
To get started though, let’s start with a quick overview of panels and how they work.
An Overview of Panels and Panel Respondents
Panel companies recruit people specifically to take surveys and participate in studies, so using panel respondents to give survey data can be beneficial for your project. It is a convenient way to find a targeted audience and it can be quick. Response rate is typically high since respondents are being paid to take the survey.
Panelists are often recruited to participate in research through advertising online, or they may be recruited through offline methods like newspapers, inserts in product packaging, at trade events, or through direct mail.
Responsible panel management companies let panelists clearly opt in and opt out at any point, and they clearly communicate their privacy policies.
The best panel providers will review and test a survey that is going out to their respondents. They’ll work with survey project owners to make sure that the survey experience is a good one, and that survey data is provided in a timely manner.
Panelists may be paid cash incentives, or they may gather points for each response or be entered in drawings. Incentives vary, and go up with the difficulty of reaching the target audience.
For instance, a general panel of people who live in the US will receive a lower incentive than a panel of oncologists in the US who have practiced for over 20 years.
The panel provider will work with you to provide information on the target audience available for your need as well as costs and timeline for survey data to be provided.
Questions and Answers on Survey Data Via Panels
And now, Bret’s insight into the panel questions that come up most often, and his tips on how to work best with panels.
Q: What’s the first thing that people running a survey with a panel should be aware of, especially if it is their first time collecting survey data using a panel?
A: It really starts with great survey design. Anyone who runs their study through a survey (whether it’s online, on paper, or a phone interview) should learn great survey design. There are particular rules to follow that all surveys, no matter the subject matter, can benefit from.
Online panels have some of the most casual survey takers. There is no interviewer, no one there to ask questions of if the survey isn’t clear, so the survey really has to stand on it’s own — it should be clear, concise, unambiguous and present zero bias.
Q: How exactly does survey design come into play?
A: Generally people are short on time nowadays. Many respondents will answer your survey taking enough time as necessary for them to answer truthfully and thoughtfully (they are paid for their time and most panel respondents are double opt-in recruited), but great survey design will help protect your survey data from those who are not as apt to apply themselves.
Great survey design techniques will help you create a survey that will minimize time spent thinking and interpreting survey questions and give respondents more time to simply react to your survey and share the feedback that you’re looking.
Some basics of great survey design are only including the questions that you really need, and making sure that everything that you do in the survey ties back to the main goal of your project.
Any issues that arise in the design phase should also be viewed through the lens of a respondent who is taking time out of their day to answer.
Q: What should I do even before I begin making my survey?
A: Definitely set goals for your survey. From the survey questions that you choose all of the way through the report types that you use after the survey data has been collected, all phases of your study should help you towards reaching your goals.
If you feel that your survey goals aren’t clear, make an initial draft of your survey. Then test by answering your survey yourself. With each question, ask if this is 100% clear to the audience that you’re targeting.
Better yet, recruit colleagues and friends to take the survey. Preferably, find people who are close to fitting in the target audience that you need and have no prior knowledge of your study.
Never hesitate to add the project time that you need to effectively plan out and test your survey.
Q: What are some common pitfalls to collecting data with panels?
A: Some common issues that I’ve run into are more annoyances than problems, but some can have permanent consequences.
An example of a more serious issue is the assumption of some panel customers that they will receive a standard report of demographics (age, gender, location, etc…) for each panel respondent who takes the survey. This is not provided generally, so if you need to know these demographics they should be added as questions to your survey.
Some panel companies have means to deliver this information AFTER each respondent has answered your survey, but as this is not a standard practice (and it can incur a “data cost” from some panel companies) you run the real risk of not ever knowing this demographic information.
Also, very important here — if you have quotas in your survey, meaning that you need a ratio of male:female respondents or need a geographically even disbursement of respondents, again, your demographic questions will have to be asked as survey questions so that your quotas are updated in real time as each panelists takes your survey.
Collect Better Data With Panels Our insider guide shows you how. Get the Ebook
Q: OK, so that’s a major pitfall, but are there any others?
A: Yes, another relates to disqualification. Always create disqualifying questions in your survey for the criteria that makes your audience unique. This will go a long way towards getting you the best survey data for your purposes.
Panel companies do their best to provide the most targeted audiences, however they may not be able to account for every demographic that you would like. In addition, it is possible for a panel respondent to have changes in their circumstances that have not yet been communicated to the panel company.
Need to capture brand feedback results from males ages 21 – 30 who purchase energy drinks? Even if the panel company is targeting these respondents directly, make sure that your survey questions do the following:
- Ask to confirm their gender.
- Ask to confirm their age.
- Ask to confirm that they do indeed purchase energy drinks.
This protects you from any possible errors in sample distribution as well.
Q: What makes for a great disqualifying question?
A: Avoid yes/no answers for your disqualifying question. In efforts to minimize the margin of receiving respondents who are not your exact target audience you don’t want to give away the purpose of your study.
Here’s an example related to energy drink use where the target audience is those who purchase energy drinks on a regular basis:
If your study begin with this question in isolation on page one, you might think to ask, “Do you purchase energy drinks?”
Respondents may guess that they have to answer “yes” in order to continue taking your survey. This opens the door to those who have only drunk this type of beverage one time in the past, perhaps a year or so ago. This person’s opinion may be of very low value to you.
A better set of disqualifying questions would be the following:
Please list the types of products that you use to supplement your energy levels:
- I consume energy bars (Powerbar, Cliff Bar, etc…)
- I drink coffee
- I consume energy gels
- I consume energy drinks (5 hour energy, Monster Energy Drink, Red Bull)
- None of the above
Then, if at least, “I purchase energy drinks is selected,” then the second part of the disqualifying question can be asked:
How often do you purchase energy drinks?
- Once every 6-12 months
- Once every several months
- Once a month
- Couple times a month
- Weekly
- Daily
Then, perhaps screen-out anyone who doesn’t drink them at least once a month. This would ensure that survey data comes from those who purchase energy drinks at least once a month.
Q: What can I do to keep the cost of my panel down?
A: Length of interview is a big cause of cost. This is a standard term used by panel companies even though “interview” still refers to an online survey. Limiting your survey questions to only those that assist with the main goals that you’ve set for your survey will help you limit cost.
Unless your subject matter is fairly heavy and warrants differently, keep your survey under 10 minutes on average for completion time.
Typically, projects are very similarly priced for any experience under 10 minutes, but for larger orders of 500 respondents or more there certainly are price differences between a 4 minute survey and an 8 minute survey.
To avoid price surprises for survey data from a panel, develop and finalize your disqualifying questions BEFORE your quoting discussion begins.
The number one factor that increases your panel order cost is the incidence rate of your survey. If your screening questions are too difficult and no one passes through to the end of your survey, then you would be alerted of a price increase as it is more difficult than expected to gather responses.
If you’re transparent about what your exact screening criteria will be from the very beginning of conversations with your panel sales rep, then there should be no surprises. If criteria change or additional requirements are added then costs will likely go up.
This also allows you some flexibility.
Let’s say that you have a quote above your budget for a set of disqualifying questions. At this point you might work with the panel company to word your disqualifying questions differently so that there will be a higher incidence rate, and therefore, a lower price for your study.
Q: What if I need to collect survey data from panel respondents several times?
A: Leting the panel company or representative know all the details of your project, including whether there will be one or more follow up surveys for the same respondents, will help make sure you can reach the audience more than once.
Q: Any other tips for working with panels?
A: Don’t ask for personal information in your surveys. Respondents are the panel companies’ resources and surveys using panel respondents will be anonymous.
Be sure that you are using a survey software and a plan level that allows for the disqualifying logic and for quotas.
Set quotas (response limits) in your survey. That way you will be sure to pay for only that amount of responses that you intend to gather.
A url variable needs to be added to your survey link to track respondents. There will be three urls that you’ll use to redirect respondents for the panel company. One will be for completed surveys, one for disqualifications and one for responses over quota.
Q: What about sample size?
A: In working to get a panel you’ll need to know how many responses you’d like to purchase. This is where calculations for sample size are needed. There are some important factors to this decision:
- Population size: the size of the entire population that you want to survey.
- Level of accuracy or margin of error: The most commonly used is 5%.
- Confidence level: or how sure you are that the sample accurately reflects the population. The most commonly used confidence level is 95%.
Following is a quick sample size list that you can use (95% @ 5%):
You may not know your population size and if that is the case 400 is a good sample size to use. Larger sample sizes can be more precise, but as you get above this mark there are diminishing returns.
Ready to Collect Survey Data With a Panel?
As you can see from our conversation with Bret, there are important factors to consider when working with panels to get great survey data. It can be a convenient and cost-effective way to reach your target audience, and with a little extra effort towards the details you can have the quality data to base your decisions on.
