Chi-square test in online surveys: testing associations

When you want to know whether two categorical variables are associated — gender and brand preference, education level and voting behavior, location and customer satisfaction — the chi-square test is the right tool. Here's the methodology with a concrete example.

In surveys we often collect categorical data — gender, age group, education level, brand preference, voting choice, place of residence. The obvious question in analysis is often: are two of these variables associated? Do men really prefer Brand A more often than women? Does place of residence correlate with voting preference? This is where the chi-square test comes in.

Categorical (nominal) vs. metric data — the crucial difference

Categorical data — often also called "nominal" — assign observations to classes: gender (male/female/non-binary), brand preference (Brand A/B/C), region of residence, color preference. When you assign numbers to these classes (man = 1, woman = 2), those numbers are pure labels, not values. "Woman = 2" is not "more" than "man = 1" — means are meaningless, "greater than" comparisons impossible. Metric data, by contrast — age, income, Likert sum scores — have real magnitude relations: "40 years" is objectively more than "30 years." Statistical tests are tailored to one of these two worlds. The t-test needs metric data (after all, it compares means). For categorical data there's a dedicated method — the chi-square test.

Unlike the t-test, which compares means, the chi-square test examines frequencies. If you've collected two variables each in categories, you can use a crosstab to calculate whether the observed frequency distributions deviate from statistical chance.

What does the chi-square test examine?

The logic is surprisingly simple: you count how often each combination of two variables actually occurs, and compare these actual frequencies with the frequencies you'd expect if the two variables were completely independent of each other. If the deviations are small, the variables are probably independent. If they're large, there's an association.

The chi-square value quantifies exactly this deviation. The larger it is, the more the observed frequencies deviate from the expected ones — and the less likely it is that this deviation arose purely by chance.

Eligibility

Before you calculate, check three points. First, both variables must be categorical (nominal or ordinal — for continuous variables, use the t-test or correlation). Second, the observations must be independent: a person may appear only once in the table. Third, no cell may have an expected frequency below 5 — otherwise, with small samples or many categories, the test becomes unreliable. Rule of thumb: at most 20% of cells may have expected values below 5, and none may be at zero or below.

If the last requirement is violated, there are fallback options: combine categories (two sparsely populated age classes become one), exact tests (Fisher's exact test for 2×2 tables), or other methods.

Step 1: Set up the crosstab

Imagine this: in a customer survey you asked 200 people whether they prefer our brand (Brand A) or the competitor brand (Brand B) — broken down by gender. The data look like this:

Beobachtet	Männer	Frauen	Zeilensumme
Marke A	52	38	90
Marke B	48	62	110
Spaltensumme	100	100	200

At first glance it looks as if men prefer Brand A and women prefer Brand B. But is this distribution systematic or simply sampling variation?

Step 2: Calculate the expected frequencies

If gender and brand preference were actually independent, the frequencies would have to split proportionally to the overall distribution. 90 of 200 respondents prefer Brand A overall — that's 45%. If gender plays no role, 45% in each gender group would have to prefer Brand A.

E_{ij} = \frac{R_i \cdot C_j}{N}Expected frequency per cell

Here Rᵢ is the row total of the cell, Cⱼ the column total, and N the grand total. For the men–Brand-A cell: (90 × 100) ÷ 200 = 45.

The complete table of expected frequencies looks like this:

Erwartet (bei Unabhängigkeit)	Männer	Frauen
Marke A	45	45
Marke B	55	55

Step 3: Calculate the chi-square value

For each cell you calculate the squared deviation of the observed from the expected frequency, divided by the expected frequency. Then you sum over all cells.

\chi^2 = \sum_{i,j} \frac{(O_{ij} - E_{ij})^2}{E_{ij}}Chi-square value

What chi-square measures — the idea behind the formula

Behind this formula lies a surprisingly simple idea. Each summand examines a single cell: how far does the actual value deviate from what we would have expected? In our table, 52 men prefer Brand A — we would have expected 45. The deviation is 7. That's the core of each cell.

Around it come two tricks. First: we square the deviation. Not to make it look bigger — but so that a deviation of +7 in one cell and one of −7 in another don't cancel each other out when we add everything up at the end. Both are real deviations, both should count.

Second: we divide by the expected frequency. Why? A deviation of 5 with an expected number of 5 is huge (twice what was expected). The same deviation of 5 with an expected number of 500 is almost nothing. This division makes deviations comparable across cells with differently sized expectations.

Then we sum everything up — and get chi-square. What does the value tell us? If it were zero, the data would fit exactly the expectation that the two variables are independent. The larger it is, the further your observations deviate from that — and the less likely it is that this is just chance.

Here O stands for the observed and E for the expected frequency per cell. In our example: (52−45)² ÷ 45 + (38−45)² ÷ 45 + (48−55)² ÷ 55 + (62−55)² ÷ 55 = 1.089 + 1.089 + 0.891 + 0.891 ≈ 3.96.

Step 4: Degrees of freedom and significance level

The calculated chi-square value alone is not yet meaningful — it must be compared with a critical value that takes two reference quantities into account.

Degrees of freedom. For a crosstab they are calculated as (number of rows − 1) × (number of columns − 1). For a 2×2 table like ours: (2−1) × (2−1) = 1 degree of freedom. For a 3×4 table it would be 6.

What degrees of freedom really are — a brief insight

"Degrees of freedom" sounds like statistics jargon — but they're quite concretely countable. The idea behind it: if the row and column totals of your table are already fixed, into how many cells can you still freely enter a number without something failing to add up at the end?

In our 2×2 table, row 1 has a total of 90, row 2 a total of 110, and both columns have 100. Write the number 10 in the top-left cell — and everything else is immediately determined. The top right must be 80 (so row 1 comes to 90), the bottom left 90 (so column 1 comes to 100), the bottom right 20 (so the rest adds up). Only a single cell was truly free to choose — that's 1 degree of freedom.

For a 3×4 table, 6 cells would remain free, the other 6 would follow automatically — hence the formula (3−1) × (4−1) = 6.

Why this matters: the chi-square distribution looks different depending on the degrees of freedom. From a table with more cells, the value can become larger purely through chance effects — so the threshold beyond which one speaks of "significant" shifts with the table size. This is exactly why the statistics table lists a separate critical value for each degree of freedom.

Significance level. Usually 5% (that is, α = 0.05), and in particularly sensitive studies 1%. This is the accepted probability of error for wrongly rejecting the independence hypothesis.

What "5%" really means — the probability in plain terms

What do these 5% mean in plain terms? Imagine you run your survey a hundred times in a row — each time with different people, but under exactly the same conditions. And suppose that, in reality, there were no genuine association at all between the two variables.

At a significance level of 5%, you would nonetheless "detect" an association in roughly 5 of these 100 runs — even though it objectively doesn't exist. Those are the random outliers we deliberately accept.

With 1% it would be only about 1 in 100 runs — the test becomes stricter, but you then also need a stronger deviation in the data for anything to become significant at all.

Which level you choose depends on the risk: for ordinary market research the usual 5% suffices; for clinical studies or legally relevant surveys you choose more conservatively.

What is the "critical value" — and where does it come from?

Where does this "critical value" actually come from? Statisticians long ago calculated a threshold for every possible combination of degrees of freedom and significance level.

The idea behind it: if a chi-square value, under the assumption of independence, would be exceeded in only 5% of all random samples, then exactly this value is the "5% threshold." If your own calculated value lies above it, we no longer consider the result to be chance — and we speak of a significant association.

Instead of recalculating this threshold for every setup, you simply look it up in the chi-square distribution table: the values have long been tabulated there for every degree of freedom and every common level. For 1 degree of freedom and α = 0.05, it reads 3.841.

If your calculated chi-square value lies above it, you belong to the 5% of all possible samples that would look this extreme — even if in reality there were no association at all. You reject the assumption of independence and knowingly accept that in exactly these 5% of all repetitions you would be wrong with your statement.

In our example: 1 degree of freedom, α = 0.05 → critical value 3.841. Our calculated chi-square value of 3.96 lies just above it. The null hypothesis "gender and brand preference are independent" is rejected. There is a statistically significant association.

Step 5: Assess the strength of the association

Statistical significance only tells you whether an association exists — not how strong it is. With large samples, even very small effects become significant. For the effect size, you use complementary measures:

Phi coefficient (for 2×2 tables) or Cramér's V (for larger tables). Values around 0.1 count as a weak association, around 0.3 as moderate, and from 0.5 on as strong. In our example, Cramér's V is roughly 0.14 — a weak association that, despite significance, should not lead to a marketing about-face.

What the effect-size values actually tell you — the idea behind them

What do these numbers actually tell us? The idea behind it: the higher the coefficient, the more one variable "reveals" about the other.

With a value of 1, if you knew only the category of the first variable, you could predict the category of the second with 100% accuracy. Put differently: all men would choose a different brand than all women, a perfect separation.

With a value of 0, knowing the first variable brings no advantage at all — the distribution of the second variable is identical in both groups, that is, pure independence.

Values in between accordingly lie between "complete information" and "no information at all." So our Cramér's V of 0.14 says: knowing the gender helps us a little in predicting brand preference — but really only a little.

Important: this statement applies to the specific sample you surveyed — not automatically to the entire population.

When the chi-square test doesn't fit

The chi-square test is limited to two categorical variables. As soon as you want to examine more than two variables at once (gender × age group × brand preference), you need log-linear models. If one of the variables is ordinal and you want to factor in the ordering (rather than ignore it), consider the Mantel-Haenszel test or ordinal regression.

And as mentioned above: with small expected frequencies, switch to Fisher's exact test. It's mathematically more elegant, but computationally more demanding to analyze — modern statistics packages handle this automatically.

Conclusion

The chi-square test is the standard tool for analyzing associations between categorical variables. It's methodologically simple, intuitively understandable, and available in any statistics software. What matters is not to treat it as the endpoint: significance alone says nothing about practical relevance. Only with a complementary effect size (phi, Cramér's V) does a complete picture emerge.

In QUESTIONSTAR you create crosstabs with two clicks right in the analysis view — the chi-square value, the p-value, and Cramér's V are calculated and displayed automatically.

Sources

Bortz, Jürgen, and Nicola Döring: Forschungsmethoden und Evaluation für Human- und Sozialwissenschaftler. 4th edition. Springer, 2006.
Agresti, Alan: An Introduction to Categorical Data Analysis. 3rd Edition. Wiley, 2018.
Field, Andy: Discovering Statistics Using IBM SPSS Statistics. 5th Edition. SAGE, 2018.