The customers easiest to reach are not always the customers a decision needs to serve. A team recruiting through its community forum may hear from knowledgeable enthusiasts while missing people who struggled once and never returned. Increasing the number recruited through the same channel can make the result more precise without making it more representative.
Sampling bias arises when coverage or selection systematically distorts the sample in ways relevant to the research question. Unequal selection probabilities are not automatically a flaw—some probability designs use them deliberately with appropriate analysis. The concern is whether the method supports the conclusions being drawn about the intended population.
Define who the findings need to describe
“All users” is often too vague to guide recruitment. A question about first-time setup may require people who have recently attempted it, including those who did not finish. A question about a specialist reporting workflow may require experienced administrators. Those populations call for different recruitment choices.
Write down the target group, the channels available and who each channel cannot reach. An in-product invitation misses people who no longer open the product. An email invitation requires a usable address and access to that account. A desktop-only study may exclude people whose mobile experience is central to the decision.
AAPOR’s best practices for survey research discuss sampling and the need to consider how a study reaches its intended population. The principle applies to product research too: describe the route into the sample before interpreting the people who arrived through it.
Match the recruitment method to the claim
Random sampling from a suitable frame can support population estimates when the design and analysis meet their assumptions. Purposive sampling selects people for relevant experience and can be valuable for exploring a process or understanding variation. The latter should not quietly acquire the claims of a probability sample because the results are shown as percentages.
Quotas can improve balance on selected characteristics, but they do not establish balance on everything that matters. Matching a customer base by age will not necessarily correct a sample made entirely of unusually engaged users. More recruitment channels may improve coverage while also introducing differences that need to be recorded.
For a fictional language-learning app, research with daily users could illuminate advanced practice routines. It would be a weak basis for explaining why new learners abandon their first lesson. The same sample can be useful for one question and unsuitable for another.
Carry the boundary into the report
Describe eligibility, recruitment, incentives and the characteristics that matter to interpretation. Use conclusions that match the evidence: a finding among active community members should be labelled accordingly. This is especially important when a memorable quotation travels further than the methods section.
Distinguish selection from non-response bias, where relevant differences arise between people who answer and those who do not. Both can affect a study, and neither is repaired simply by making the final sample larger.
When the missing group could change the decision, investigate it directly where feasible. If it cannot be reached, make the uncertainty part of the decision rather than disguising it with a broad label such as “customer feedback”.