What is Statistical Significance?

A result is statistically significant when it would be sufficiently unlikely to occur by chance alone, under the assumption that no real effect exists, for researchers to treat it as evidence of something genuine. The machinery works by contradiction. You posit a null hypothesis (the boring world: no difference between versions, no relationship between variables), collect data, and compute a p-value: the probability of observing data at least as extreme as yours if the null were true. A small p-value says your data would be surprising in the boring world, which counts as evidence against it. If p falls below a pre-chosen threshold, the significance level, conventionally 0.05, the result is declared significant.

Read that definition carefully, because every popular misreading violates it. The p-value is not the probability the null hypothesis is true, not the probability your finding is a fluke, and not a measure of how large or important the effect is. It is a statement about how surprising your data would be in a hypothetical world. Nothing more, and often quite a lot less than decision-makers assume.

A Brief History

The p-value tradition descends from Ronald Fisher's work in the 1920s, where 0.05 was offered as a convenient benchmark rather than handed down as law, a historical accident that hardened into ritual. Jerzy Neyman and Egon Pearson built the rival hypothesis-testing framework (explicit alternatives, error rates, power) that textbooks later blended with Fisher's into the hybrid most practitioners now use without knowing its seams. By the 2010s, concern about misuse had grown loud enough that the American Statistical Association took the unusual step of issuing a formal statement on p-values (2016), spelling out what they do and do not mean, a document provoked substantially by significance-chasing's role in science's replication crisis.

The Concepts Around the Number

Significance level (α). Your false-positive budget: testing at 0.05 means accepting a 1-in-20 chance of "detecting" an effect that isn't there. Nothing is sacred about 0.05. Costly decisions deserve stricter thresholds; cheap reversible ones can tolerate looser.

Statistical power. The probability of detecting an effect that does exist, driven mostly by sample size and effect size. Underpowered studies fail silently: they return "not significant" for real effects, and teams misread that as "no difference". Absence of evidence gets dressed up as evidence of absence.

Effect size and confidence intervals. Significance answers "is it distinguishable from zero?"; effect size answers "how big is it?"; a confidence interval answers both at once by showing the plausible range. With enough traffic, a 0.1% conversion lift will be exquisitely significant and commercially irrelevant. Statistical significance is not practical significance, and the confusion between them is the field's most expensive habit.

Multiple comparisons. Test twenty metrics at α = 0.05 and chance alone will hand you roughly one "significant" result. Every extra metric, segment, and peek at interim results spends the same false-positive budget, which is why disciplined teams pre-register their primary metric and treat everything else as exploratory.

Significance in Product Research

The concept earns its keep wherever product teams compare numbers: A/B tests, survey comparisons across segments or waves, and benchmark studies tracking metrics like the System Usability Scale release over release. The recurring sins are equally predictable: stopping an A/B test the moment the dashboard turns green (peeking inflates false positives dramatically), declaring victory on a secondary metric when the primary one stayed flat, and slicing data into segments until something clears the bar. Equally important is knowing where the machinery does not belong. A five-user usability test is a diagnostic instrument, not a statistical one, and demanding p-values from it misunderstands both. Small-sample qualitative work earns its confidence through convergence and repetition, not through α.

How to Use It Honestly

1. Choose the primary metric, the threshold, and the sample size before launching, and write them down.
2. Run to the planned sample; resist the green light at day two.
3. Report effect size and confidence interval alongside p. The decision lives in the magnitude, not the asterisk.
4. Treat unplanned subgroup findings as hypotheses for the next study, not conclusions from this one.
5. Remember that "not significant" means "this study didn't detect it", not "there is nothing there".

The Benefits

For all the abuse it attracts, significance testing does real work: it imposes a shared, pre-agreed standard of evidence, it protects organisations from reorganising around noise, and it forces the discipline of stating hypotheses before seeing outcomes. In high-volume experimentation cultures it is the referee that keeps a hundred simultaneous tests from generating a hundred false victories.

The Limitations

The framework answers a question subtly different from the one decision-makers ask; it is mute about effect size; its binary significant/not-significant framing manufactures false certainty on both sides of an arbitrary line; and it is gameable, consciously or not, through peeking, metric-shopping, and subgroup-mining. None of these are flaws in the mathematics. All of them are flaws in how the mathematics meets human incentives, which is exactly where research discipline has to live.

The Takeaway

Statistical significance is a filter against being fooled by randomness, a good and necessary thing, but it was never a verdict on truth or importance. Use it as one input: fix your hypotheses in advance, mind your power, read the effect size, respect the multiple-comparisons budget, and let practical significance make the final call. The teams that internalise this get the benefit the machinery was built for: fewer confident mistakes.