What is Research Bias?

Research bias is any systematic error that causes a study's results to deviate from the true value in a consistent direction, arising from the design, conduct, analysis, or reporting of the research rather than from chance. The contrast with random error is the key to the whole subject: random error makes results imprecise and shrinks with sample size, which is what margins of error price; systematic error makes results wrong, and grows more confident with sample size, because a larger biased sample is simply a more precise estimate of the wrong number. Every research method in this glossary can be read as a set of defenses against specific biases, and this entry organizes them by where in a study each bias enters.

Biases of Who You Studied

Sampling bias: the frame or the draw systematically excludes or over-includes parts of the population. Non-response bias: those who answer differ from those who don't. Attrition bias: those who leave differ from those who stay. Survivorship bias: reasoning only from the cases that made it (current customers, completed projects) while the departed carry the lesson. Each is a selection story, and each is invisible in the data it leaves behind.

Biases of How You Asked and Watched

Response bias in its many forms: social desirability, acquiescence, yes-no framing effects, and the anchoring and order effects that instrument craft exists to manage. Moderator and interviewer bias: the researcher's questions and reactions shaping the answers. Observer effects: behavior changed by being watched. Recall bias: memory reconstructing rather than retrieving, worst for frequency and duration questions, mitigated by in-the-moment capture.

Biases of How You Analyzed

Confirmation bias: seeing the expected pattern, coding toward it, remembering the sessions that fit. Analytic flexibility: metric shopping, subgroup hunting, and outcome switching, the p-hacking family, defended by pre-registration. Data-handling bias: exclusions and outlier decisions made after seeing results, defended by pre-set cleaning rules. Overgeneralization: claims stretched beyond the sample's reach.

Biases of How You Reported

Publication and reporting bias: positive, tidy findings presented; null, messy, and disconfirming ones filed away, so that the organization's body of evidence skews toward what worked. Framing bias: the same finding presented as 90% success or 10% failure, steering the decision it informs. Sponsor bias: research commissioned to confirm a decision, and reported as if it inquired.

The General Defenses

Bias cannot be eliminated, only anticipated and bounded, and the defenses recur across the families. 1. Decide before you look: hypotheses, metrics, exclusions, and analyses pre-committed. 2. Randomize and blind wherever the design allows, for selection and expectancy. 3. Standardize instruments and moderation, so the researcher is not an uncontrolled variable. 4. Triangulate across methods with different biases, so agreement means something. 5. Seek the disconfirming case on purpose. 6. Keep the audit trail and report the limitations, the sample, and the exclusions with every finding. 7. Build the defenses into defaults (unmoderated formats that remove the moderator, templates with neutral wording, screeners on behavior) so they operate when attention lapses.

What to Remember

Research bias is the consistent lean that more data only makes more confident: in who was studied, how they were asked and watched, how the data was analyzed, and how the findings were told. Map each study against the four families before fielding, pre-commit what can be pre-committed, randomize and blind what can be, triangulate, hunt the disconfirming, and report the limits. No study is unbiased; a good one knows which way it leans and says so.