Sampling Bias

Sampling bias occurs when the people in your study differ systematically from the people you are trying to understand. Your findings describe your sample accurately—but your sample does not describe your target population.

Common Sources

• Self-selection: People who volunteer for research are inherently different from those who do not. They tend to be more engaged, more opinionated, and more comfortable sharing feedback
• Channel bias: Recruiting via social media reaches social media users. Recruiting via an app's feedback button reaches users who noticed and clicked the button—your most engaged segment
• Survivor bias: Testing only current users ignores everyone who already churned. Your "user feedback" reflects people who stayed, not the ones you lost
• Convenience sampling: Testing with whoever is available—colleagues, friends, nearby people—produces results that describe your immediate network, not your users

Managing Sampling Bias

You cannot fully eliminate sampling bias, but you can reduce it:

• Diversify channels: Recruit from multiple sources to avoid over-representing a single segment
• Screen for diversity: Track demographics and ensure your sample is not accidentally homogeneous
• Acknowledge limitations: State clearly who your sample represents and who it does not

The biggest risk is not having sampling bias—it is pretending you do not.