Introduction
Hypothesis testing is the structured procedure for deciding whether the result of an experiment, such as the difference between two variants in an A/B test, is strong enough evidence to reject the assumption that nothing changed. For product teams it is the machinery behind every "variant B won" verdict, and the discipline that stops a two-day dashboard bump from becoming a roadmap decision. This article walks the procedure as product teams actually use it, flags the steps where experiments quietly go wrong, and shows what a test result can and cannot decide.
What is Hypothesis Testing?
In product research, hypothesis testing is the procedure that turns an experiment into a decision: you state what you expect a change to do (variant B will increase checkout completion), you state the boring alternative (it makes no difference), you run the experiment to a planned sample, and a pre-agreed statistical rule decides whether the observed difference is too large to be chance. The pieces are covered in their own entries: the default claim of no effect that the test tries to reject, the p-value and threshold that make up statistical significance, and the interval that says how big the effect might be. What this entry adds is the procedure as a whole, in the order a team runs it, because the order is where the honesty lives: decide before you look, or the data will decide for you.
The Procedure, Honestly Run
1. State both hypotheses before touching data.
The null (H₀: the variants convert equally) and the alternative (H₁: they differ), including whether the alternative is two-sided (any difference) or one-sided (a difference in a specified direction). One-sided tests buy sensitivity at the price of blindness to the other direction, and choosing one after seeing which way the data leans is a classic cheat.
2. Choose the test and the threshold.
The test follows the data's shape: a t-test for comparing means, a chi-square for comparing proportions and categories, and so on; each carries assumptions worth checking rather than assuming. The significance level (α) is the false-positive budget, conventionally 0.05, properly set by the cost of being wrong in this decision, not by convention alone.
3. Plan the sample for the effect that matters.
Power analysis, before fielding: how many observations to have a strong chance of detecting the smallest effect worth acting on. Skipping this step is how teams run three-day A/B tests, find nothing, and conclude nothing was there.
4. Collect to plan, then compute.
Run to the planned sample. Stopping early because the dashboard looks good inflates false positives dramatically, and metric-shopping after the fact spends the α budget invisibly, the confirmation bias failure modes the procedure exists to prevent.
5. Decide, then translate carefully.
Reject or fail to reject, and then say what that means in effect-size language: the direction, the magnitude, the confidence interval. "Statistically significant" is the start of the interpretation, never its end.
Where It Lives in Product Research
Every A/B test is hypothesis testing with a UI, and the framework equally underwrites survey comparisons (did satisfaction move between waves? do segments differ?), benchmark tracking (is this quarter's SUS genuinely higher?), and experiment-shaped product questions generally. It is equally important to know where it does not belong: five-participant usability sessions are diagnostic, not inferential, and demanding p-values of them misreads both traditions. Small-sample qualitative work earns confidence through convergence and repetition, not through α.
The Benefits
The procedure forces claims into testable form before evidence arrives, sets a shared standard for what counts as enough, quantifies the risk of being fooled by noise, and (followed honestly) protects organizations from reorganizing around randomness. Its paper trail (stated hypotheses, planned samples, pre-chosen metrics) is also the best available audit against wishful analysis.
The Limitations
Everything cataloged under significance applies: the binary verdict flattens evidence, the p-value answers a subtler question than the one asked, big samples make trivial effects "significant" and small ones hide real effects, and the machinery is gameable through peeking, shopping, and slicing. Hypothesis testing decides whether chance is a plausible explanation; whether the effect matters, and what to do about it, remain human judgments it cannot make.
In Short
Hypothesis testing is a contract with your future self: name the claim, the rival, the test, the threshold, and the sample before the data can tempt you, then honor the contract and translate the verdict into magnitudes people can act on. Run that way, it is the most reliable argument-settler research owns; run loosely, it is a random-number generator with academic manners.
Further reading
For the procedure in full:
Articles:
1. Hypothesis Testing: A Step-by-Step Guide - Scribbr
The five-step procedure with worked examples, test-selection guidance, and the assumptions behind the common tests.
2. A Refresher on Statistical Significance - Harvard Business Review
The decision-maker's translation layer: what a test verdict does and does not license you to conclude.