What is a Null Hypothesis?

The null hypothesis (written H₀) is the default claim a statistical test sets out to challenge: that there is no effect, no difference, no relationship. Its counterpart, the alternative hypothesis (H₁), is the claim the researcher usually cares about: the redesign changes conversion, the segments differ, the variables are related. The machinery of testing is deliberately asymmetric. You assume the null, compute how surprising your data would be in that boring world (the p-value), and reject the null only if the surprise crosses a pre-set threshold. What you never do, strictly speaking, is prove the null. A test that fails to reject it has returned a shrug, not a confirmation.

Why Reason Backwards?

The indirect logic descends from Ronald Fisher's work in the 1920s, and his most famous illustration remains the best. A colleague, Muriel Bristol, claimed she could taste whether milk had been poured before or after the tea. Fisher's move was to assume she couldn't (the null), design an experiment (eight cups, four each way), and calculate how improbable a perfect performance would be under pure guessing. Guessing all eight correctly has odds of 1 in 70; when she did exactly that, the null became untenable. The structure generalises: science and product analytics alike make progress by specifying what chance alone would produce, then checking whether reality outperforms it. (Bristol, the record suggests, passed the test.)

The Two Ways to Be Wrong

The framework names its own failure modes. A Type I error rejects a true null: you declare an effect that isn't there, a false positive whose probability is capped by your significance level (α, conventionally 0.05). A Type II error fails to reject a false null: a real effect goes undetected, usually because the sample was too small for the effect's size, a study's statistical power being its defence. The two trade off, and choosing α is choosing which mistake you'd rather risk. Shipping a harmless tweak on a false positive is cheap; rebuilding an onboarding flow on one is not, and the threshold should reflect that.

Nulls in Product Research

Every A/B test is a null-hypothesis test wearing a dashboard: H₀ says the variants convert identically, and the tooling computes the evidence against it. Survey comparisons across segments or waves work the same way. The framework's discipline is mostly about honesty in advance: state H₀ and H₁ before looking, fix the primary metric, and size the sample for the smallest effect worth detecting. The recurring misreadings are equally predictable. "Not significant" gets reported as "no difference" (a shrug promoted to a verdict), and a rejected null gets reported as a large or important effect, which it may not be. The concepts explored under statistical significance and confidence intervals exist precisely to keep those translations honest.

The Benefits

The null-hypothesis framework forces claims into testable form, provides a shared standard of evidence across teams and fields, and quantifies the risk of being fooled by randomness. Its asymmetry is a feature: by making "nothing is happening" the position that must be defeated, it puts the burden of proof on the exciting claim, which is where the burden belongs.

The Limitations

The null is almost never exactly true (two versions of anything differ by something), so with enough data every null falls, significance included, importance not guaranteed. The binary reject/fail-to-reject verdict flattens a continuum of evidence into a yes/no. And the framework tempts ritual: nulls chosen because they're easy to reject rather than meaningful to test. It answers one narrow question well (could chance alone plausibly produce this?) and every other question badly.

The Takeaway

The null hypothesis is the skeptic your analysis has to convince: the standing assumption that nothing interesting is happening. Use it properly by stating it in advance, powering your study to detect effects that matter, and translating its verdicts carefully, remembering that failing to reject is not confirming, and rejecting is not measuring. The boring world is the benchmark; beating it is only the beginning of the argument.