Research can go wrong in a consistent way. A study reaches only enthusiastic customers, asks them leading questions and reports the most favourable answers. More responses may make its results look increasingly stable while leaving the underlying distortion intact.
Research bias refers to systematic error arising from how a study is designed, conducted, analysed or reported. It differs from random variation, although both can affect the same result. A narrow confidence interval addresses particular forms of uncertainty under a model; it does not certify the recruitment, measurement or interpretation that produced the estimate.
Locate the problem in the study process
Start with selection. Sampling bias can arise when coverage or recruitment misrepresents the intended population. Non-response bias concerns relevant differences between those who answer and those who do not. Attrition creates a related concern when people leave a study over time.
Then examine measurement. Wording, response options, recall and the study environment can affect answers. A moderator’s prompts can influence a task, while a tracking event can fail to represent the behaviour its label suggests. A recorded number is not automatically a sound measure.
The Cochrane Handbook’s discussion of risk of bias addresses systematic error in trial results. Its specific assessment framework belongs to that setting, but the distinction between a plausible source of bias and proof of its actual effect is useful more broadly. Identifying a risk does not tell you exactly how much a result has shifted.
Check the choices made after collection
Analysis can favour a conclusion through selective exclusions, outcome changes or repeated searches for a favourable subgroup. Exploration can generate worthwhile questions, but presenting a discovery made after many comparisons as though it were the sole planned test misrepresents the evidence.
A fictional onboarding experiment might improve one engagement measure while leaving completion unchanged. Reporting only engagement makes the result sound broader than it is. Keep the planned outcomes, departures and relevant negative findings visible so that the decision can reflect the whole study.
Cognitive bias can also shape qualitative interpretation. A compelling quotation may receive more attention than contradictory accounts. Reviewing the dataset systematically and preserving a route back to the source makes that choice easier to examine.
Match the safeguard to the mechanism
A better sampling frame will not repair a leading question. Neutral wording will not repair missing participants. Predefined analysis choices will not repair a broken measurement. The useful response begins with the part of the process that could have produced the distortion.
Plan consequential measures and exclusions in advance, pilot the instrument, document recruitment and check relevant alternatives. Use randomisation or masking where they suit the design, and consider triangulation with sources whose limitations differ. None of these steps guarantees a study free of bias.
Report the remaining limits in language that affects the conclusion. “Among the active users we recruited” is narrower than “customers”, and sometimes that boundary is precisely what makes a finding trustworthy. A useful account of bias explains what could change the decision and which additional evidence would help resolve it.