Introduction
Ice cream sales and drowning deaths rise together every summer, and no one proposes banning ice cream. Yet the same reasoning error, treating things that move together as things that cause each other, ships in product decks constantly: users who do X retain better, so make everyone do X. Correlation versus causation is the most consequential distinction in applied analytics, and the gap between them is where confounders, reverse causation, and expensive mistakes live. This article maps the ways correlation misleads, and the designs that earn genuine causal claims.
The Distinction
Correlation means two variables move together: when one is higher, the other tends to be too (or lower, for negative correlation). Causation means one actually produces change in the other: intervene on X and Y responds. Correlation is a fact about observed data; causation is a claim about what would happen under intervention, and the second never follows from the first alone. Any observed association has at least four possible explanations: X causes Y; Y causes X (reverse causation); some third factor Z causes both (confounding); or the pattern is coincidence, the noise that significance testing exists to price. Deciding among them is the entire art.
The Classic Failure Modes
Confounding. The summer heat drives both ice cream sales and swimming (hence drownings): a lurking variable manufactures the association. Product analytics is a confounder farm, because users are not randomly assigned to behaviours; the users who adopt a feature differ from those who don't in motivation, expertise, and need, and those differences (not the feature) may drive the retention gap.
Reverse causation. "Users who complete onboarding retain better" reads as onboarding causing retention; it reads equally well as already-committed users being the ones who bother to finish onboarding. The arrow's direction is an assumption until a design pins it.
Selection effects. Correlations computed on filtered populations (survivors, completers, respondents) can invert or invent relationships that don't exist in the whole, the analytic cousin of sampling bias.
Coincidence at scale. Comb enough metric pairs and spectacular correlations appear by chance; entire websites collect them for laughs. Dashboards comb thousands of pairs daily.
Earning Causal Claims
1. Randomise where you can.
Random assignment is the confounder killer: when chance decides who gets the treatment, treated and untreated groups match on everything else in expectation, and a difference in outcomes has one live explanation. This is the entire authority of the A/B test and the controlled experiment: not fancier measurement, but assignment by coin-flip.
2. Approximate experiments where you can't.
When randomisation is impossible (pricing for existing customers, features already shipped), quasi-experimental methods (before/after with comparison groups, staged rollouts, natural experiments) recover partial causal leverage, with honesty about what they can't rule out.
3. Interrogate observational correlations before acting.
Ask the four-explanations question explicitly. Check temporal order (causes precede effects). Hunt plausible confounders and control for the measurable ones, knowing statistical control only reaches what you measured. Look for dose-response patterns and mechanism: epidemiology's Bradford Hill considerations, ported to product data.
4. Use qualitative work to find the mechanism.
Correlations propose; mechanisms convince. Interviews and observation reveal why a behaviour might drive an outcome, turning a pattern into a testable causal story, which a subsequent experiment can settle: the full mixed-methods loop.
The Practical Discipline
Language is the first control: "associated with", "predicts", and "correlates" are earned by observation; "drives", "causes", and "increases" are earned by design. Readouts that respect the vocabulary keep organisations honest, because the sentence "power users who enable notifications churn less" and the sentence "notifications reduce churn" recommend entirely different investments, and only one of them is supported by a correlation. When the causal question matters enough to fund action, it matters enough to fund the experiment.
The Takeaway
Correlation is a clue; causation is a conclusion, and the road between them runs through design: randomise where possible, approximate where not, interrogate confounders and direction everywhere, and let mechanisms from qualitative work guide which experiments to run. The ice cream never caused the drownings. The question for every correlated metric pair on your dashboard is: what's the summer?
Further reading
For the reasoning and the remedies:
Articles:
1. Correlation vs. Causation - Scribbr
The distinction with examples of confounding, directionality problems, and the designs that support causal claims.
2. A Refresher on Statistical Significance - Harvard Business Review
The companion discipline: pricing coincidence before arguing about causes.