What are Outliers?

An outlier is an observation that lies unusually far from the bulk of the data: extreme enough that it plausibly comes from a different process than the rest. That "different process" framing is the useful one, because it names the fork every outlier presents. Some extremes are errors: a timer that kept running after a participant walked away, a decimal slip, a bot in the survey sample. Others are genuine: the power user whose behaviour really is tenfold, the accessibility scenario where the flow truly collapses, the niche use case that predicts next year's market. Errors should be fixed or removed; genuine extremes are evidence, and the whole craft is telling them apart before any statistics run.

Why They Matter So Much

Small samples give extremes enormous leverage: one forty-minute struggle in a twelve-person study drags the mean task time far from anything typical (the reason medians anchor usability reporting), a single point can own a fitted regression line, and variance-based machinery (t-tests, standard errors, correlations) inherits the distortion. In product analytics the stakes flip: revenue and usage distributions are long-tailed by nature, and the "outliers" are often the business, which is why trimming the top of a revenue distribution to make the chart prettier is analysis malpractice of a different kind.

Detecting Them

1. Look first.
Plots beat rules: histograms, box plots, and scatterplots reveal extremes, clusters of extremes, and the skew that explains them, before any threshold gets argued.

2. Apply conventional fences, as flags not verdicts.
The interquartile-range rule (beyond 1.5×IQR from the quartiles, the box plot's whisker convention) and z-score screens (beyond roughly 3 standard deviations, itself unreliable in small or skewed samples) are standard first passes. They nominate candidates; they decide nothing.

3. Investigate the story behind each candidate.
Session recordings, timestamps, and open-text answers usually reveal which fork you're on: the "outlier" whose video shows a doorbell mid-task is an interruption artefact; the one showing genuine confusion at step three is your finding. This is where mixed instruments pay off, since a Ballpark task result arrives with its recording attached, and diagnosis takes minutes instead of guesswork.

Handling Them Honestly

Fix verified errors; remove with a log. Documented, criterion-based exclusion (decided ideally before fielding, as part of data cleaning) is legitimate; quiet deletion because a point spoiled the result is p-hacking's cruder cousin.

Keep genuine extremes, and manage their leverage. Report medians and IQRs alongside means; use transformations (log time is standard for durations) or robust methods where extremes are structural; and run the analysis with and without the disputed points, reporting both when conclusions differ, which is itself a finding about fragility.

Mine them qualitatively. Extreme cases are sampling gold: the users at the edges stress the product in ways the middle never will, and a follow-up interview with an outlier often teaches more than ten with the median.

The Takeaway

Outliers are forks, not nuisances: error or evidence, decided by investigation rather than by whichever answer flatters the analysis. Detect them visually, screen them conventionally, diagnose them with the recordings, document every exclusion, and protect the genuine extremes, because in skewed realities the edge cases are frequently the point. The rule that survives every situation: nothing gets deleted silently.