What is the Chi-Square Test?

The chi-square (χ²) test evaluates categorical data by comparing the counts you observed against the counts you would expect if a null hypothesis were true, summing the squared discrepancies (scaled by the expected counts) into one statistic whose size measures how far reality strayed from the null's prediction. Large discrepancies across the table produce a large χ² and a small p-value, and the standard machinery converts that into a verdict. Where the t-test compares means of continuous measures, chi-square lives entirely in the world of categories and counts, which makes it the survey researcher's and product analyst's constant companion.

The Two Classic Uses

Goodness of fit. One categorical variable against a claimed distribution: do sign-ups spread evenly across weekdays? Do answer choices match last year's proportions? The null supplies the expected split; the test asks whether the observed counts deviate more than sampling noise allows.

Test of independence. Two categorical variables in a contingency table: does plan tier relate to churn status? Does variant relate to conversion? The null says the variables are unrelated (each cell's expected count is what row and column totals alone would produce), and a significant result says the pattern of association is unlikely to be chance. This is the textbook analysis behind every "did the segments answer differently?" crosstab, and the classical test behind many A/B conversion comparisons (where large-sample proportion tests and chi-square give equivalent answers).

The Fine Print

1. Counts, not percentages.
The test runs on raw frequencies; feeding it percentages breaks the mathematics. Convert back before testing.

2. Independent observations.
Each count must be a separate unit: two hundred answers from twenty repeat participants violate the assumption and inflate confidence, the same independence rule that governs every test in this family.

3. Expected counts need minimum size.
The standard guidance: expected counts below about five in a meaningful share of cells make the approximation unreliable; merge sparse categories or use exact alternatives (Fisher's exact test for small 2×2 tables).

4. Significance is not strength.
With large samples, trivial associations turn significant; pair the verdict with an effect-size measure (Cramér's V is the usual companion) and read the cell-level residuals to see where the table diverges, the part stakeholders actually care about. The general cautions of statistical significance apply undiluted.

5. Association is not causation.
A significant independence test says the variables travel together in this sample; the explanations still span the full casebook, from causation to confounding to selection.

Chi-Square in Product Research

The everyday applications: comparing answer distributions across segments or survey waves; testing whether task success (a category: succeeded, partial, failed) differs between prototypes in a comparative study; checking whether a recruited sample's category mix matches the population's (a goodness-of-fit audit of representativeness); and validating that randomisation balanced categorical traits across experiment arms. In each, the test's gift is discipline: it prices the shuffle before anyone narrates the pattern.

The Takeaway

The chi-square test is the counting world's referee: observed against expected, discrepancy priced in probability. Feed it independent raw counts, respect the small-cell limits, attach an effect size, and read the residuals for the story. A century and a quarter after Pearson, it remains the fastest honest answer to the most common analytical question there is: is this table telling me something, or just shuffling?