What is Hypothesis Testing?

Hypothesis testing is the structured statistical procedure for deciding whether data provides sufficient evidence against a default claim. The full apparatus assembles pieces covered in their own entries: a null hypothesis (no effect, no difference) faces an alternative; a test statistic summarises the data; a p-value measures how surprising that statistic would be if the null were true; and a pre-set significance threshold converts the p-value into a decision, with all the caveats that statistical significance carries. What this entry adds is the procedure as a whole: the sequence, the choices inside it, and the discipline that makes the output mean what it claims to mean.

The Procedure, Honestly Run

1. State both hypotheses before touching data.
The null (H₀: the variants convert equally) and the alternative (H₁: they differ), including whether the alternative is two-sided (any difference) or one-sided (a difference in a specified direction). One-sided tests buy sensitivity at the price of blindness to the other direction, and choosing one after seeing which way the data leans is a classic cheat.

2. Choose the test and the threshold.
The test follows the data's shape: a t-test for comparing means, a chi-square for comparing proportions and categories, and so on; each carries assumptions worth checking rather than assuming. The significance level (α) is the false-positive budget, conventionally 0.05, properly set by the cost of being wrong in this decision, not by convention alone.

3. Plan the sample for the effect that matters.
Power analysis, before fielding: how many observations to have a strong chance of detecting the smallest effect worth acting on. Skipping this step is how teams run three-day A/B tests, find nothing, and conclude nothing was there.

4. Collect to plan, then compute.
Run to the planned sample. Stopping early because the dashboard looks good inflates false positives dramatically, and metric-shopping after the fact spends the α budget invisibly, the confirmation bias failure modes the procedure exists to prevent.

5. Decide, then translate carefully.
Reject or fail to reject, and then say what that means in effect-size language: the direction, the magnitude, the confidence interval. "Statistically significant" is the start of the interpretation, never its end.

Where It Lives in Product Research

Every A/B test is hypothesis testing with a UI, and the framework equally underwrites survey comparisons (did satisfaction move between waves? do segments differ?), benchmark tracking (is this quarter's SUS genuinely higher?), and experiment-shaped product questions generally. It is equally important to know where it does not belong: five-participant usability sessions are diagnostic, not inferential, and demanding p-values of them misreads both traditions. Small-sample qualitative work earns confidence through convergence and repetition, not through α.

The Benefits

The procedure forces claims into testable form before evidence arrives, sets a shared standard for what counts as enough, quantifies the risk of being fooled by noise, and (followed honestly) protects organisations from reorganising around randomness. Its paper trail (stated hypotheses, planned samples, pre-chosen metrics) is also the best available audit against wishful analysis.

The Limitations

Everything catalogued under significance applies: the binary verdict flattens evidence, the p-value answers a subtler question than the one asked, big samples make trivial effects "significant" and small ones hide real effects, and the machinery is gameable through peeking, shopping, and slicing. Hypothesis testing decides whether chance is a plausible explanation; whether the effect matters, and what to do about it, remain human judgments it cannot make.

The Takeaway

Hypothesis testing is a contract with your future self: name the claim, the rival, the test, the threshold, and the sample before the data can tempt you, then honour the contract and translate the verdict into magnitudes people can act on. Run that way, it is the most reliable argument-settler research owns; run loosely, it is a random-number generator with academic manners.