Causal research asks what difference something makes. If customers who complete onboarding are more likely to stay, the association does not show that increasing onboarding completion will improve retention. Those customers may already have stronger reasons to use the product.
The causal question concerns a comparison with what would have happened under another condition. That unobserved alternative is the counterfactual, and a research design needs a credible way to reason about it.
Specify the change and the outcome
In an illustrative onboarding study, define the intervention precisely: a guided setup flow, an invitation to a training session, or access to a new support channel. Each could affect different people through different mechanisms.
Define who the result concerns, what outcome will be measured, and over what period. “Improve retention” is incomplete until the team specifies the relevant unit, activity, and time window. Distinguish the effect of offering an intervention from the effect among people who actually use it; selecting only users of the intervention can reintroduce differences the study was meant to address.
The government’s Magenta Book explains experimental, quasi-experimental, and theory-based approaches to evaluating effects. The appropriate approach depends on the question and the conditions in which evidence can be gathered.
Examine the comparison’s credibility
Random assignment can make groups comparable in expectation, supporting a causal estimate when implementation and analysis are sound. It does not guarantee identical groups in a particular sample or prevent missing data, spillovers, and measurement problems.
When randomisation is unavailable, a credible quasi-experimental approach may use a threshold, a comparison trend, or another feature of how an intervention occurred. The assumptions need to be stated and examined; adding many variables to a model is not an automatic solution to confounding.
A simple before-and-after comparison is vulnerable to other changes occurring over time. Seasonality, customer mix, and unrelated releases may explain some or all of an apparent improvement.
Explain the effect and its scope
Report the estimated difference and uncertainty, alongside evidence about how the intervention was delivered. Qualitative research can help investigate mechanisms and circumstances without turning an interview account into a numerical causal estimate.
Consider whether the result is likely to apply elsewhere through generalizability. A successful intervention in one audience or period may depend on conditions that do not travel. A useful causal account makes those conditions visible, so the next decision can draw on more than a headline saying the test worked.