The t-Test in Online Surveys: Comparing Means

The t-test checks whether a measured mean differs from an expected value, or whether two means differ from each other — methodologically clean and with a manageable amount of calculation. Here are the three most important variants, each with a concrete example.

The t-test is one of the most widely used statistical procedures in quantitative social research. It checks whether an observed difference between means is random or systematic. In market research, in employee surveys, in clinical studies — wherever two groups or a target-vs-actual comparison is on the table, the t-test is the obvious first choice.

Three variants are relevant in practice: the one-sample t-test (a mean against an expected value), the unpaired two-sample t-test (two independent groups), and the paired two-sample t-test (one group before/after). We'll go through them one by one.

What does the t-test actually check?

At its core, the t-test answers a simple question: is the difference we see in the data large enough not to be explained by chance? Samples vary — even if the population were identical, two randomly drawn groups would show slightly different means. The t-test gives a probability that the measured difference is "just variation" and not a "real effect."

The logic behind it is hypothesis testing: we formulate a null hypothesis ("no difference"), calculate how likely our data would be if the null hypothesis were true, and decide on the basis of this probability whether to reject the null hypothesis.

One-sample t-test: mean against target value

The one-sample t-test checks whether the mean of a single sample deviates significantly from a given expected value. Use cases in surveys: benchmark comparison (does our employee satisfaction deviate from the industry mean?), target-vs-actual comparison (does our CSAT score reach the defined target?), comparison against historical references (is the NPS in this wave above last year's median?).

Concrete example: your management has set a goal that employees should identify with the company strategy on a scale from 1 to 5 — target mean 4.0. In this year's employee survey with 80 participants, the mean comes out at 3.7. Question: is the difference of 0.3 a genuine warning sign, or can it be explained by sampling variation?

The calculation runs in three steps. First, you calculate the mean of your sample and the standard deviation. Then you plug these values into the t-test formula:

t = \frac{\bar{x} - \mu_0}{s \,/\, \sqrt{n}}t-test (one-sample)

The numerator is the difference between the sample mean (x̄) and the expected value (μ₀). The denominator is the standard deviation (s) divided by the square root of the sample size (n). The result is the t-value — a standardized measure of the size of the deviation relative to the variation in the data.

What the t-test really compares — the idea behind the formula

It sounds like mathematics — but the idea is simple. The numerator (x̄ − μ₀) is the raw deviation: by how much does your measured mean deviate from the expected value? In our example: 3.7 instead of 4.0 — so 0.3.

The denominator (s ÷ √n) is what's called the standard error of the mean. It answers the question: how much would the mean fluctuate by chance alone if you repeated the same survey several times with different people? With high variation in the data and a small sample, this fluctuation is large. With uniform answers and a large sample, it's small.

By dividing the numerator by the denominator, we compare the observed deviation with the fluctuation we'd normally expect. So the t-value says: in how many standard errors does the difference we found lie? A value of 2 means: twice as far away as we'd expect by chance alone — already fairly suspicious. A value near 0 says: everything within the normal range of variation, no reason to worry.

Third, you compare the calculated t-value against a critical value from the t-distribution table. You'll find this critical value in any statistics formula collection — it depends on two quantities: the degrees of freedom (sample size minus 1) and the chosen significance level (usually 5% or 1%).

What do these 5% actually mean in plain terms?

Imagine you run your survey a hundred times in a row — each time with different people, but under exactly the same conditions. And suppose that in reality there were no difference at all between the sample mean and the expected value.

With a significance level of 5%, you would nonetheless "detect" a deviation that looks statistically significant in about 5 of these 100 runs — even though it objectively does not exist. Those are the random outliers we deliberately accept.

With 1% it would be only about 1 in 100 runs — the test becomes stricter, but you then also need a stronger deviation in the data for anything to become significant at all.

Which level you choose depends on the risk: for an internal employee or customer survey the usual 5% is enough; for clinical studies or legally relevant surveys you choose more conservatively.

Where does this "critical value" actually come from?

Statisticians long ago calculated a threshold — the critical value — for every possible combination of degrees of freedom and significance level.

The idea behind it: if, under the assumption of "no difference," a t-value would be exceeded in only 5% of all random samples, then that value is precisely the "5% threshold." If your calculated value lies above it, we no longer regard the result as chance — and we speak of a significant deviation.

Instead of recalculating this threshold for each setup, you simply look it up in the t-distribution table: the values are long since tabulated there for every degree of freedom and every common level.

If your calculated t-value (in absolute terms) lies above the table value, you belong to the 5% of all possible samples that would look this extreme — even if in reality there were no difference at all. You reject the null hypothesis and deliberately accept that you'd be wrong in exactly those 5% of all repetitions.

If the absolute value of your calculated t lies above the critical value, the deviation is significant — the null hypothesis "the identification value equals the target of 4.0" is rejected. If it lies below, the deviation is compatible with chance — no escalation needed.

What degrees of freedom mean in a t-test — a brief insight

In a t-test, "degrees of freedom" means something different than in a crosstab — but the idea stays similar. If the mean of your sample is already fixed, how many of your individual measured values could you still freely change without the mean tipping over?

A small example: you have five measurements with a mean of 12. Say four of the five values are 10, 12, 11, and 13. Then the fifth value must be exactly 14 — otherwise the mean doesn't come out. So one "freedom" is used up by the fact that we know the mean. With n values, therefore, n − 1 degrees of freedom remain.

In a two-sample comparison, two means are calculated from the data — so n₁ + n₂ − 2 degrees of freedom remain. In the paired test we work with the differences per person, and their mean is the only parameter — so again n − 1, where n is the number of pairs (not the individual measured values).

Why this matters: the t-distribution looks different depending on the degrees of freedom. With small samples it's wider — the uncertainty is greater, so you need a stronger deviation before we speak of "significant." With large samples it becomes narrower and approaches the normal distribution — the threshold drops. That's exactly why the statistics table has its own row with different critical values for every degree of freedom.

Unpaired two-sample t-test: two independent groups

The unpaired two-sample t-test compares the means of two independent groups — that is, groups whose members have nothing to do with each other. This is the most common variant in surveys: men vs. women, customers vs. non-customers, location A vs. location B.

Example: you want to know whether customer satisfaction differs between your two branches in Berlin and Munich. You survey 120 customers per location and compare the means. Berlin: 4.1 out of 5. Munich: 3.8 out of 5. Is this difference of 0.3 systematic or sampling noise?

The formula is a direct extension of the one-sample case. The numerator is the difference between the two group means; the denominator is a weighted combination of both variations:

t = \frac{\bar{x}_1 - \bar{x}_2}{s_p \cdot \sqrt{\dfrac{1}{n_1} + \dfrac{1}{n_2}}}t-test (unpaired, equal variances)

Here sₚ is the so-called pooled standard deviation — the weighted average of the standard deviations of both groups:

s_p = \sqrt{\frac{(n_1 - 1) s_1^2 + (n_2 - 1) s_2^2}{n_1 + n_2 - 2}}Pooled standard deviation

Important: the unpaired t-test assumes that the variances of the two groups are similar (Levene's test or visual check). If the variances differ strongly, use the Welch correction, which uses the separate variations instead of the pooled standard deviation — many statistics packages do this automatically.

The degrees of freedom here are the sum of the two sample sizes minus 2 (n₁ + n₂ − 2). The rest of the analysis — calculate the t-value, pull the critical value from the table, compare — works identically to the one-sample case.

Paired two-sample t-test: the same people before and after

The paired t-test is conceptually the most elegant: it doesn't compare two different groups, but the same group at two points in time. Before-after studies, pre-test/post-test designs, A/B tests with repetition — wherever each measurement can be matched to an earlier measurement.

Example: you've introduced a new onboarding program for employees. Before the introduction, 60 new employees rated their onboarding experience on 1 to 5 (mean 3.2). After the program reform and six months of use, you repeat the survey with the same group of people (mean 3.9). Question: has the perception improved significantly, or is the difference noise?

Statistically, the paired t-test is actually a one-sample test on the differences: for each person you calculate the difference between the before and after value (d = x_after − x_before) and then check whether the mean of these differences deviates from zero. The formula:

t = \frac{\bar{d}}{s_d \,/\, \sqrt{n}}t-test (paired)

Here d̄ is the mean of all individual differences, s_d the standard deviation of the differences, and n the number of pairs. The degrees of freedom are n − 1 (as in the one-sample case), because here n is not the number of observations but the number of difference values.

This construction is methodologically cleaner than two independent tests, because the intra-personal differences are automatically factored out — a person who generally rates more critically compensates against themselves. The statistical power is therefore usually higher than in an unpaired test with the same number of cases.

Judging the strength of the difference — Cohen's d

Statistical significance only tells you whether a difference exists — not how large it is. With very large samples, even tiny differences that are practically meaningless become significant. That's why, alongside the p-value, you always also report the effect size. For the t-test, the usual measure is Cohen's d:

d = \frac{\bar{x}_1 - \bar{x}_2}{s_p}Cohen's d (effect size for the t-test)

The numerator is the difference of the means, the denominator the pooled standard deviation. Unlike with the t-value, here you don't divide by √n — Cohen's d is independent of the sample size and describes the pure size of the effect. Cohen proposed the following rules of thumb: d ≈ 0.2 = small effect, d ≈ 0.5 = medium effect, d ≈ 0.8 or larger = strong effect. Important: these are conventions, not laws of nature — in many fields other thresholds apply.

What Cohen's d actually says — the idea behind it

Behind Cohen's d lies an intuitive question: how large is the difference between the groups, measured in "standard deviations of individual people"?

At d = 0 the means lie exactly on top of each other — no effect. At d = 1 the difference is as large as one standard deviation of the data: a person from group A lies on average one whole typical unit of variation higher than a person from group B. That's a lot.

Put differently: a small d-value means that the distributions of the two groups overlap strongly — you cannot reliably assign an individual person to one group or the other just because you know their value. A large d-value means strong separation — the groups are clearly distinguishable as distributions.

In the Berlin-vs-Munich example above: 4.1 vs 3.8 with a typical standard deviation of about 1.0 gives d ≈ 0.3 — a small to medium effect. The difference is real (provided it's significant), but it doesn't justify a fundamental restructuring of the locations.

Eligibility — when the t-test fits and when it doesn't

The t-test is robust, but not universal. There are three prerequisites you should check before applying it.

First, the level of measurement. The variable under investigation must be interval-scaled (that is, distances are interpretable — temperature, weight, income; Likert sum scores are considered sufficient). Pure frequencies or nominal values (gender, product preference) are unsuitable — here the Chi-square test is the right partner.

Second, the distribution. The values should be approximately normally distributed. With a sufficiently large sample (n > 30 per group), the central limit theorem applies and the normality assumption may be relaxed. With small samples that have a skewed distribution, you're better off reaching for non-parametric alternatives — Mann-Whitney U for the unpaired comparison, Wilcoxon for the paired one.

Third, independence of observations. If you survey 30 employees of one department and 30 employees of another department, the observations within a department are not independent (shared supervisors, shared working conditions). Such cluster effects distort the t-test — multilevel analysis would be the clean way.

Conclusion

The t-test is a tool for one concrete question: is the difference in means that I see statistically sound? In that specificity it's very good. But as soon as you want to compare categorical variables, examine relationships between several variables, or analyze more complex study designs, you need other procedures. Here the Chi-square test belongs in the toolbox beside it — it takes over where the t-test leaves off.

In QUESTIONSTAR you calculate t-tests directly in the analysis view — the t-value, p-value, degrees of freedom, and an interpretation note appear automatically as soon as you select two means to compare.

Sources

Bortz, Jürgen, and Christof Schuster: Statistik für Human- und Sozialwissenschaftler. 7th edition. Springer, 2010.
Field, Andy: Discovering Statistics Using IBM SPSS Statistics. 5th edition. SAGE, 2018.
Schumann, Siegfried: Repräsentative Umfrage. 7th edition. De Gruyter Oldenbourg, 2019.