A test result can cross a statistical threshold without becoming useful, and miss that threshold without showing that nothing happened. Statistical significance is a judgement made by comparing a p-value with a specified significance level under a particular statistical model. It is one part of interpreting evidence, alongside the effect estimate, uncertainty and study design.
The distinction matters whenever a dashboard offers a simple winner. A small change measured precisely may be commercially unimportant; a potentially substantial change measured imprecisely may remain unresolved. The label alone cannot tell the team which situation it faces.
Understand what the p-value conditions on
A p-value describes the probability, assuming the specified null model and its other assumptions, of a test statistic at least as extreme as the one observed. Small values indicate incompatibility between the data and that model. They do not give the probability that the null hypothesis is true or that the result was caused by chance.
The American Statistical Association’s statement on p-values explains these distinctions and warns against basing conclusions solely on whether a threshold is crossed. A result below 0.05 has not acquired a universal guarantee of truth, just as one above it has not proved the absence of an effect.
For a correctly calibrated test, a significance level such as 0.05 controls a long-run false-rejection rate under the null and the test’s assumptions. That is different from saying that a particular significant result has a 5% probability of being false.
Read the size and uncertainty before deciding
Imagine a fictional experiment in which a revised search interface produces a small increase in successful searches. The interpretation needs the estimated change and a confidence interval, together with any effects on errors, response time or other consequential outcomes.
A narrow interval around a negligible change and a wide interval spanning harm and benefit call for different decisions, even if neither result crosses a chosen threshold. If the aim is to establish that two experiences are sufficiently similar, plan an equivalence or non-inferiority analysis with a justified margin rather than interpreting a non-significant difference as proof of equality.
The relevant effect need not be a difference from zero; the null hypothesis depends on the question. Keep that hypothesis visible so readers understand what the test actually assessed.
Protect the analysis from repeated opportunities to win
Testing many outcomes or subgroups creates more opportunities for false positives. Repeatedly inspecting a fixed-sample test and stopping when it becomes significant can also invalidate the intended error control. Sequential methods can support interim decisions when designed for that purpose; they are not interchangeable with unrestricted checking.
Specify primary outcomes, comparisons and stopping rules in advance. Treat unplanned analyses transparently and use suitable multiplicity handling where required. Document exclusions and tracking failures before a favourable result obscures their importance.
In an A/B test, the final decision should connect statistical evidence to the practical consequences of the change. Significance can help assess compatibility with a model. It cannot repair biased measurement, establish value to customers or decide how much uncertainty the organisation should accept.