Introduction
The gold standard says randomise, and reality frequently says no: you cannot coin-flip who gets the price change, the policy, or the feature that already shipped to everyone in Germany. Quasi-experimental design is the toolkit for exactly this territory: studies that manipulate or exploit an intervention without randomised assignment, recovering causal leverage through design cleverness instead. This article covers the main designs, what each can and cannot rule out, and how staged rollouts turned product teams into accidental quasi-experimentalists.
What is Quasi-Experimental Design?
A quasi-experiment studies the effect of an intervention without randomly assigning who receives it: the treatment happens (by policy, timing, geography, threshold, or choice), and the design's job is to construct a credible answer to the counterfactual question a randomised control would have answered automatically: what would have happened without it? The prefix marks the missing ingredient. Without randomisation, treated and untreated groups can differ systematically before the intervention (selection), and every pre-existing difference is a rival explanation the design must argue away rather than dissolve, the central threat to internal validity that separates this family from true experiments.
The Main Designs
Nonequivalent comparison groups. Compare a treated group with an untreated one that wasn't randomly formed (the region that got the rollout versus the one that didn't), ideally with before-and-after measures on both. The before-measures are the design's honesty check: groups tracking together pre-intervention make post-intervention divergence more credible.
Difference-in-differences. The workhorse formalisation: compare each group's change, not their levels, so stable pre-existing differences cancel. Its load-bearing assumption is parallel trends (absent treatment, both groups would have moved alike), checkable in the pre-period and never fully provable.
Interrupted time series. One group, many measures before and after: does the metric's trajectory break at the intervention? Strong when the series is long and the break is sharp; vulnerable to anything else that happened at the same moment.
Regression discontinuity. When treatment follows a threshold (accounts above a usage cutoff get the feature), compare units just either side of the line, who are plausibly near-identical except for treatment. Locally powerful, elegant, and silent about effects far from the cutoff.
Quasi-Experiments in Product Work
Product organisations run quasi-experiments constantly, often without the label: staged rollouts by geography or cohort (nonequivalent groups with a calendar), pricing changes applied to new customers only (a natural comparison against existing ones), feature launches evaluated against their own history (interrupted time series wearing a dashboard), thresholds in plans and limits (discontinuities waiting to be analysed). The upgrade from anecdote to inference is mostly discipline: choose the comparison before the results, check pre-trends, log everything else that changed in the window, and report the design's specific vulnerabilities alongside its estimate. Where randomisation is feasible, prefer it; where it isn't, a well-built quasi-experiment beats both intuition and a naive before/after by miles.
The Craft Rules
1. Name the counterfactual explicitly.
Every quasi-design is an argument about what would have happened otherwise; write that argument down and attack it yourself before others do.
2. Invest in the pre-period.
Baselines and pre-trends are the currency of credibility: without them, difference-in-differences is just differences.
3. Hunt co-interventions.
Marketing pushes, seasonality, pricing tweaks, and news events that coincide with the treatment are the classic assassins of time-series claims; a change log is cheap insurance.
4. Triangulate with mechanism.
Quasi-estimates gain enormously from a mechanism story: qualitative work and behavioural detail showing how the effect operates convert a contested number into a coherent finding, the standard move.
5. Scope the language.
Report as "consistent with a causal effect of roughly X, under these assumptions", not as an A/B verdict. The honesty costs a clause and buys the claim its durability.
The Takeaway
Quasi-experimental design is causal inference for the world as it is: no coin-flip, so the counterfactual must be built (comparison groups, differenced changes, interrupted series, threshold neighbours) and defended assumption by assumption. Done with pre-trends, change logs, and scoped language, it turns the rollouts and thresholds every product already has into evidence, which is the next best thing to the experiment you couldn't run.
Further reading
For the designs and their assumptions:
Articles:
1. Quasi-Experimental Design - Scribbr
The main designs with examples, and what each requires to support causal reading.
2. Correlation vs. Causation - Scribbr
The threats (confounding, selection, reverse paths) every quasi-design is engineered to answer.