What is a Confidence Interval?

A confidence interval is a range of values, computed from sample data, constructed to contain the true population value with a stated long-run success rate, conventionally 95%. Instead of reporting "42% of respondents preferred the new layout", you report "42%, 95% CI [37%, 47%]": the sample's best estimate, plus an honest account of the precision the sample size can support. The interval does double duty. It quantifies uncertainty, and it carries the information a significance verdict compresses away: how big the effect might plausibly be, in both directions.

What 95% Actually Means

The famous subtlety: the 95% describes the procedure, not the particular interval. If you repeated the study many times, each time computing an interval the same way, about 95% of those intervals would capture the true value. Any single interval either contains the truth or doesn't; the probability language belongs to the long run of the method. The everyday reading ("there's a 95% chance the truth is in this range") is technically a different kind of statement, and statisticians wince at it, but as a practical intuition it points decision-makers in a broadly sensible direction. The genuinely dangerous misreadings are elsewhere: treating the point estimate as exact, or treating values at the interval's edge as impossible.

Reading Intervals Like a Practitioner

Width is the message. A narrow interval says the sample earned its precision; a wide one says the data are stretched thin. "Conversion improved by 4 points, CI [0.5, 7.5]" and "improved by 4 points, CI [3.6, 4.4]" contain the same headline and completely different amounts of knowledge.

Check what the interval includes. An interval for a difference that spans zero is the graphical twin of a non-significant test, but it says more: it shows how large an effect remains plausible despite the non-significance. An interval hugging zero says "probably nothing here"; a wide interval spanning zero says "we don't know yet", and the difference matters for what you do next.

Three levers set the width. Sample size (the big one: quadruple the sample to halve the interval), variability in the data, and the confidence level you demand (99% intervals are wider than 95%). This is also the practical answer to "how many participants do we need?": enough to make the interval narrow enough for the decision at hand.

Intervals in Product Research

Confidence intervals belong on nearly every number research produces: survey percentages, A/B test lifts, task-success rates from usability studies, SUS scores, NPS. They are especially valuable at the small sample sizes real product work involves, where they perform their most useful public service: displaying how little a proportion from twelve participants actually pins down, before anyone builds a roadmap on it. Reporting "9 of 12 completed the task (75%, CI roughly 45% to 92%)" reframes the conversation from false precision to honest evidence. The same habit disciplines comparisons: two versions whose intervals overlap heavily are a weaker story than their point estimates suggest, and the machinery of significance testing exists for judging exactly those cases.

The Benefits

Intervals communicate estimate and uncertainty in one object, keep small samples honest, translate directly into decision terms (is the whole plausible range acceptable?), and resist the binary thinking that significance verdicts encourage. They also make research more comparable: two studies' intervals can be inspected side by side in a way two p-values cannot.

The Limitations

Intervals inherit every flaw of the data beneath them: a biased sample yields a precisely stated wrong answer, and no interval fixes sampling or response problems. The long-run interpretation is genuinely unintuitive and gets mangled in translation. And width can be manipulated by quietly lowering the confidence level, which is why the level should always be stated alongside the range.

The Takeaway

A confidence interval is an estimate wearing its honesty on the outside. Report one with every headline number, read the width before the midpoint, and let the plausible range, not the point estimate, drive the decision. Research that ships its uncertainty earns more trust than research that hides it, and deserves to.