Sample size matters, but less than most people think. Once you reach the size you need for your confidence level and margin of error, adding more responses barely improves accuracy. Who’s in your sample matters more. A large sample that leaves out part of your audience is still biased.
Early in a survey project, teams often ask, “How do we get responses from everyone?” Unless you’re working with a very small group, you don’t need to. You need responses from a smaller, well-chosen portion of your population. So the better questions are “What’s a good sample size?” and “Does our sample represent the people we care about?”
This guide covers both. You’ll find a sample size calculator, a quick reference table and a checklist for building a representative sample.
Key takeaways
- Six common types of selection bias can skew results, and most come down to who you can and can’t reach.
- For most large populations, about 385 responses gives you 95% confidence with a ±5% margin of error.
- Past roughly 10,000 people, your required sample size barely grows, no matter how big the population gets.
- A large sample can’t fix bias. Where your sample comes from matters more than how big it is.
What is a good sample size?
A good sample size is the number of responses you need to reach your chosen confidence level and margin of error. For large populations, that’s about 385 responses at 95% confidence and ±5%. Smaller populations need fewer, and a tighter margin of error needs more.
Three terms drive the calculation:
- Confidence level is how sure you can be that your results reflect the full population. Most research uses 95%.
- Margin of error (also called the confidence interval) is how far your results might land from the true value. A ±5% margin means a 60% result likely falls between 55% and 65% in the full population.
- Population size is the total number of people in the group you want to understand.
Use the calculator below to find your target sample size.
The calculator assumes a response distribution of 50%, which gives the most conservative (largest) sample size.
Here’s what the calculator returns at 95% confidence, ±5% margin of error and a 50% response distribution:
| Population size | Sample size needed |
|---|---|
| 100 | 80 |
| 500 | 218 |
| 1,000 | 278 |
| 10,000 | 370 |
| 100,000 | 383 |
| 1,000,000+ | 384 |
Notice how the numbers level off. Surveying a population of 1 million takes barely more responses than surveying 10,000. That’s why chasing a huge sample often adds cost without improving your results much. For more on why this happens, see our [introduction to sampling distributions].
Avoiding selection bias in your survey responses
When not every member of your target population has an equal chance of being chosen to take your survey, you’re at risk of polluting your data with selection bias.
There are four common ways that this occurs:
Bias through convenience samples
Convenience samples are just what they sound like: choosing respondents that we can conveniently reach without regard to their demographic data.
These samples include respondents who are easier to select or who are most likely to respond; they will not be representative of harder-to-select individuals.
Samples from online panels are a good example of convenience samples.
These panels are composed of individuals who have expressed interest in participating in surveys, leaving out individuals who may be part of the target population but are not available for interviewing through the panel.
Selection bias via undercoverage
Undercoverage happens when we fail to include all of the target population in the sampling frame.
Many online panels work hard at avoiding undercoverage bias, but the fact remains that certain demographics are underrepresented in panels.
For example, it is difficult to field online studies targeted at the total Hispanic population in the US without using a hybrid data collection approach that allows us to reach unacculturated Hispanics, who are usually underrepresented in most online panels.
Coverage bias is also common in phone surveys.
Many of these surveys use telephone list sampling frames that exclude households without landline access.
As more households substitute cell phones for their landlines, obtaining representative samples of certain demographic groups is almost impossible without including cell phone lists in the sampling frame.
Nonresponse and selection vias
Selection bias also happens when we fail to obtain responses from everyone in the selected sample.
Nonrespondents tend to differ from respondents, so their absence in the final sample makes it difficult to generalize the results to the overall target population. This is why the design of a survey is far more important than the absolute sample size to get a representative sample of the target population.
Final three sources of sample bias
Three other common ways that sample bias can creep into a survey are:
- Judgment Sample: This is a sample selected based on “representative” criteria that are chosen based on prior knowledge of the topic or target population. An example would be a study looking for a sample of teenagers, and trying to intercept them at a cross-section near a high school.
- Misspecification: This happens when we intentionally or unintentionally use screening criteria that leave out important subgroups of the population.
- Poor Data Collection: An example of this includes allowing whoever is available in the household to take the survey instead of the intended member based on certain screening criteria.
Giving reference to sample source, not size
So, when it comes to getting a representative sample, sample source is more important than sample size.
If you want a representative sample of a particular population, you need to ensure that:
- The sample source includes all the target population
- The selected data collection method (online, phone, paper, in person) can reach individuals that represent that target population
- The screening criteria truly reflect the target population and do not inadvertently exclude valuable subpopulations
- You can minimize nonresponse bias with good survey design, incentives, and the appropriate distribution method
- There are quality controls in place during the data collection process to guarantee that designated members of the sample are reached
Conclusion
The right sample size helps you get you to a reliable result. The right sample gets you to a true one. Use a calculator to set your target, then put most of your energy into who you’re reaching. That means a sampling frame that covers your whole audience, screening criteria that don’t quietly filter people out and a survey design that keeps people answering.
Alchemer Survey helps you get both parts right. You can build surveys that are easy to finish, distribute them across the channels your audience uses and track responses as they come in, so you’ll know early if a group is missing. The result is feedback that reflects your full audience, not just the people who were easiest to reach.
FAQ
What is a good sample size for a survey?
For most large populations, about 385 responses gives you a 95% confidence level with a ±5% margin of error. Smaller populations need fewer responses, and a tighter margin of error needs more. For a ±3% margin, plan on roughly 1,070 responses.
Does a bigger sample always mean better results?
No. Past a certain point, extra responses add cost without improving accuracy much. A bigger sample also can’t fix bias. If your sample leaves out part of your audience, collecting more responses from the same source only repeats that gap.
What’s the difference between sample size and a representative sample?
Sample size is how many people respond. A representative sample is one whose makeup matches your target population. You need both, but representativeness depends on where your sample comes from and how you collect it, not just how big it is.
What is a sampling frame?
A sampling frame is the list or source you draw your sample from, such as a customer email list or a panel. If your sampling frame leaves out part of your target population, your sample can’t represent that group, however many responses you collect.
What is selection bias in surveys?
Selection bias happens when some members of your target population have a lower chance of being included in your sample. Common causes include convenience sampling, undercoverage, nonresponse, judgment sampling, misspecified screening criteria and poor data collection.
How do you reduce nonresponse bias?
Keep surveys short and easy to complete, send them through the channels your audience uses, offer incentives where appropriate and follow up with nonrespondents. Comparing early and late responders can also help you spot whether nonrespondents might differ from the people who answered.