Outliers

What are Outliers in Research Data?

An outlier is an observation far enough from the bulk of the data that it plausibly came from a different process than the rest, and in product research the "different process" fork shows up constantly. Some extremes are artifacts: a session timer that kept running while a participant took a call, a bot in the survey sample, a decimal slip in an export. Others are real: the power user whose usage genuinely is tenfold, the accessibility scenario where a flow truly collapses, the enterprise account whose behavior predicts next year's segment. Artifacts get fixed or removed with a note; real extremes are evidence, often the best in the study, and the whole craft is telling the two apart before any statistic runs. Small samples make the question urgent: one forty-minute struggle in a twelve-person usability test owns the mean task time, which is why usability reporting anchors on medians and why every outlier decision belongs in the write-up.

Why They Matter So Much

Small samples give extremes enormous leverage: one forty-minute struggle in a twelve-person study drags the mean task time far from anything typical (the reason medians anchor usability reporting), a single point can own a fitted regression line, and variance-based machinery (t-tests, standard errors, correlations) inherits the distortion. In product analytics the stakes flip: revenue and usage distributions are long-tailed by nature, and the "outliers" are often the business, which is why trimming the top of a revenue distribution to make the chart prettier is analysis malpractice of a different kind.

Detecting Them

1. Look first.
Plots beat rules: histograms, box plots, and scatterplots reveal extremes, clusters of extremes, and the skew that explains them, before any threshold gets argued.

2. Apply conventional fences, as flags not verdicts.
The interquartile-range rule (beyond 1.5×IQR from the quartiles, the box plot's whisker convention) and z-score screens (beyond roughly 3 standard deviations, itself unreliable in small or skewed samples) are standard first passes. They nominate candidates; they decide nothing.

3. Investigate the story behind each candidate.
Session recordings, timestamps, and open-text answers usually reveal which fork you're on: the "outlier" whose video shows a doorbell mid-task is an interruption artifact; the one showing genuine confusion at step three is your finding. This is where mixed instruments pay off, since a Ballpark task result arrives with its recording attached, and diagnosis takes minutes instead of guesswork.

Handling Them Honestly

Fix verified errors; remove with a log. Documented, criterion-based exclusion (decided ideally before fielding, as part of data cleaning) is legitimate; quiet deletion because a point spoiled the result is p-hacking's cruder cousin.

Keep genuine extremes, and manage their leverage. Report medians and IQRs alongside means; use transformations (log time is standard for durations) or robust methods where extremes are structural; and run the analysis with and without the disputed points, reporting both when conclusions differ, which is itself a finding about fragility.

Mine them qualitatively. Extreme cases are sampling gold: the users at the edges stress the product in ways the middle never will, and a follow-up interview with an outlier often teaches more than ten with the median.

Where This Leaves You

Outliers are forks, not nuisances: error or evidence, decided by investigation rather than by whichever answer flatters the analysis. Detect them visually, screen them conventionally, diagnose them with the recordings, document every exclusion, and protect the genuine extremes, because in skewed realities the edge cases are frequently the point. The rule that survives every situation: nothing gets deleted silently.