---
url: "https://questionstar.com/blog/statistical-tests/"
title: "Which statistical test should I use? A guide with examples"
description: "Difference, relationship or change? Four questions lead you to the right statistical test: a decision path, an overview of 14 cases and worked examples."
lastmod: "2026-11-12"
type: "blog"
---

# Which statistical test should I use? A guide with examples

*I have survey data and a question about it. Which statistical test should I use?*

Difference, relationship or change? Four questions lead you to the right statistical test: a decision path, an overview of 14 cases and worked examples.

In short: four things decide which statistical test fits. First, what you want to know: a difference between groups, a relationship between two variables, a change in the same people, or the precision of a single value. Second, the level of measurement of your variables: categories, ordered categories, or numbers with equal distances. Third, the number of groups. Fourth, whether the groups are independent, that is, made up of different people whose answers are unrelated to one another, or paired, that is, two measurements of the same people (Malhotra and Birks 2007, pp. 493, 525-526). For the most common cases in surveys, this means: the appropriate t-test compares two means, the chi-square test in a cross-tabulation compares frequencies in categories for independent groups, and correlation tests the relationship between two measured quantities.

This article is a guide. It leads you to the right method and from there to the article that explains it step by step. In this order:

1. [Why test at all?](#why-test)
2. [Which four questions decide the test?](#four-questions)
3. [Which test goes with which data? The overview](#overview)
4. [What does this look like in examples?](#two-groups)
5. [What do I need to check before I test at all?](#before-testing)
6. [What does QUESTIONSTAR calculate itself, and how do I calculate the rest?](#questionstar)

The typical situation: the survey is closed, the report shows means and percentages for each location, and someone asks whether the differences are "real." Of the dozens of methods in a statistics program, a handful is usually enough for this. The examples come from an invented training provider with locations in Hamburg, Cologne and Munich. All numbers are made up and deliberately kept small.

## Why test at all?

The report says: satisfaction in Hamburg 4.06, in Cologne 3.54, on a scale from 1 to 5. Hamburg is ahead. You can see that without any calculation. Why isn't that enough?

Flip a coin ten times. You get exactly five heads and five tails in only about one in four sets of ten flips. In the other three, one side comes up more often, even though nothing is wrong with the coin. So six heads surprises nobody. Ten heads is a different matter: a fair coin lands on the same side ten times in only about 2 out of 1,000 sets of ten flips. It isn't impossible. But anyone who sees it starts to doubt the coin.

Survey data are similar. Only twelve people were surveyed at each location. Twelve other people would have answered a little differently, and the mean would have been different. So a difference between two groups also appears when the customers at both locations are in fact equally satisfied, just as one side usually comes up more often with a fair coin. So the fact that Hamburg is ahead proves nothing yet. The real question is: is the difference larger than what chance alone usually produces? Is it more like the "six heads" or more like the "ten heads"?

That is exactly what a statistical test works out. It starts by assuming that there is in fact no difference, just as you start by assuming a fair coin. Then it determines how likely a difference at least this large would be under that assumption, by chance alone. This probability is called the **p-value**.

- **If the p-value is small,** as with the ten heads, the assumption "no difference" fits the data poorly. You doubt it, just as you doubt the coin.
- **If the p-value is large,** as with the six heads, the difference may well be chance. That doesn't prove the locations are the same: even a slightly biased coin often produces only six heads. The data just aren't enough to demonstrate a difference.

You decide where "small" begins before the analysis. The usual threshold is 0.05, that is, 5 in 100 (Schumann 2019, p. 206). If the p-value is below it, the result is called "statistically significant."

In one respect, survey data are harder than the coin. With the coin, you know in advance what chance usually produces. With survey data, the test first has to work that out from your data: from how many people answered and how much their answers differ. How that is done depends on the data. The calculation is different for means than for proportions, for two groups than for three, for the same people than for different people. That is why there isn't one test, but many.

In short: every statistical test answers the same question: could the result also be explained by chance alone? The answer is the p-value. A small p-value is evidence against chance alone, a large one leaves the question open. Whether a difference is large enough to matter is something the test doesn't tell you. You can read more about the p-value and its limits in the article [p-value and statistical significance](/blog/statistical-significance/). Which test goes with your data follows from four things.

## Which four questions decide the statistical test?

In my course "Principles of Survey Research," choosing the test statistic is the third of five steps of a hypothesis test ([slide 185](/principles-of-survey-research/chapter-5/slide-185.html)). Before that comes the question you want to answer.

**1. What do you want to know?** In surveys, it is almost always one of these questions:

- **Difference:** Are customers in Hamburg more satisfied than customers in Cologne?
- **Relationship:** Is likelihood to recommend related to how long someone has been a customer?
- **Change:** Do the same people rate their skills higher after the course than before?
- **Precision of a single value:** How precise is "65% would book again"?

**2. What is the level of measurement of the variables?** What matters is whether a variable represents categories (nominal: location, format, yes/no), an order (ordinal: a single question with five response options) or numbers with equal distances (metric, that is, interval or ratio: age, duration, a mean of several rating questions that is treated as metric). Tests that work with means, such as the t-test, require metric variables (parametric tests). Tests that need only frequencies or ranks work with nominal or ordinal data (non-parametric tests) (Malhotra and Birks 2007, p. 525). The article [Levels of measurement](/blog/levels-of-measurement/#likert) explains how to determine the level of a question and why single Likert items are disputed.

**3. How many groups are you comparing?** One, two or more. What counts is how you split the data for the analysis, not how you collected them. Women and men from the same survey are two samples for the comparison (Malhotra and Birks 2007, p. 493).

**4. Are the groups independent or paired?** Samples are independent when each group contains different people and their answers are unrelated to one another, both across the groups and within each group. That is the case for customers in Hamburg and Cologne who were surveyed individually. Different people alone aren't enough: if, say, married couples or several members of the same team answer, their answers are related, and the analysis has to account for that. Samples are paired when both measurements come from the same respondents: the same people before and after the course, or the same people rating two offers (Malhotra and Birks 2007, pp. 525, 531). To check your data, ask: can you match every answer in one series to the same person's answer in the other? Then the data are paired, and you need a test that uses these pairs. The section [The same people answer twice](#paired) shows why that is and what goes wrong otherwise.

The following diagram translates these four things into six yes/no questions.

The path leads to the usual method, not to the only possible one. If you want to be careful with the assumptions, choose the method with less restrictive assumptions (Malhotra and Birks 2007, p. 492).

## Which statistical test for which data? The overview

The table summarizes the decision path. The column "How to calculate it" names, for each method, a route that is sure to work with your data: the report in QUESTIONSTAR, or a calculation with exported data or an AI assistant. How the second route works is described [further down](#your-data).

| Your question | Level of measurement | Groups | Test | How to calculate it | Read more |
|------|------|------|------|------|------|
| How precise is a proportion? | nominal (yes/no) | one | confidence interval for a proportion | proportion and number of responses from the report, interval with the calculator | P-value and statistical significance, Confidence interval |
| Does a mean differ from a target value? | metric | one | one-sample t-test | in the report: t-test (one sample) | The t-test, one-sample case |
| Do two groups differ in their means? | metric outcome variable | two, independent | independent-samples t-test | in the report: t-test (independent) | The t-test |
| Do two groups differ on a question with ordered categories? | ordinal outcome variable | two, independent | Mann-Whitney U test | with exported data or an AI assistant | Rank tests |
| Do proportions or distributions differ between groups? | both nominal (or ordinal, with the order ignored) | two or more, independent | chi-square test in the cross-tabulation | in the "Crosstab" report | Cross-tabulation, The chi-square test |
| Do three or more groups differ in their means? | metric outcome variable | three or more, independent | one-way analysis of variance (ANOVA) | with exported data or an AI assistant | section below |
| Do three or more groups differ on a question with ordered categories? | ordinal outcome variable | three or more, independent | Kruskal-Wallis test | with exported data or an AI assistant | section below |
| Has the mean of the same people changed? | metric | two, paired | paired t-test | in the report: t-test (paired) for two questions of the same survey, also via MCP | The t-test |
| Have the same people changed on a question with ordered categories? | ordinal | two, paired | Wilcoxon signed-rank test (for the assumption, see the section) | with exported data or an AI assistant | section below |
| Has a yes/no proportion changed in the same people? | nominal (yes/no) | two, paired | McNemar test | with exported data or an AI assistant | section below |
| Are two categorical variables related? | both categorical (with the order ignored for ordinal variables) | n/a | chi-square test, strength with Cramér's V | in the "Crosstab" report | Cross-tabulation |
| Are two variables with ordered categories related? | both ordinal, or one ordinal and one metric | n/a | Spearman's rank correlation (Kendall's tau if there are many tied values) | in the report: correlation, also via MCP | section below, Levels of measurement |
| Are two measured quantities related? | both metric | n/a | Pearson correlation | in the report: correlation, also via MCP | section below, Correlation and causation |
| By how much does one quantity change when another increases? | metric dependent variable | n/a | regression analysis | in the report: regression, also via MCP | section below, Correlation and causation |

*My own illustration based on Malhotra and Birks 2007, p. 494 (Figure 17.4), p. 526 (Figure 18.9) and p. 536 (Table 18.19). The table of test statistics in the course ([Marx 2026, slide 192](/principles-of-survey-research/chapter-5/slide-192.html)) follows the same order. The slide presents its example as one sample with two variables, while this article presents it as a comparison of two groups. Both use the same chi-square test. Column "How to calculate it": as of November 2026.*

## Which statistical test for two groups?

### Means: the independent-samples t-test

- **When:** Two groups with different people, one variable treated as metric, and a question about the difference between the means (Malhotra and Birks 2007, pp. 525, 528).
- **Assumption:** Independent observations, approximately normally distributed values or sufficiently large groups, similar spread in both groups (Malhotra and Birks 2007, pp. 527, 529). If the spread is unequal, use Welch's t-test, see [The t-test](/blog/the-t-test/#voraussetzungen).
- **Typical mistake:** Treating paired data as independent, or testing all pairs separately when there are three or more groups.

**Example.** Overall satisfaction is calculated as the mean of four questions, each rated on a five-point scale from "very dissatisfied" (1) to "very satisfied" (5). Twelve people each from Hamburg and Cologne answered.

| Location | n | Mean | Standard deviation |
|------|------|------|------|
| Hamburg | 12 | 4.06 | 0.44 |
| Cologne | 12 | 3.54 | 0.44 |

The difference is 0.52 scale points. The t-test gives t = 2.90 with 22 degrees of freedom, p = 0.008. This is how you read the three figures:

- **t = 2.90:** The observed difference is almost three times as large as the variation you would expect from chance alone with two groups of twelve respondents. This is the benchmark from the coin flip: what chance usually produces. The further t is from zero, the less well the difference fits chance alone.
- **22 degrees of freedom:** a figure for the amount of data. Once the mean of twelve answers is fixed, only eleven answers are free. The twelfth follows from them. With two groups, that makes 11 + 11 = 22. The number belongs in the report because the p-value depends on it: the same t is less convincing with few answers than with many.
- **p = 0.008:** If both locations were in fact the same, a difference this large or larger would occur in fewer than one in a hundred such samples. That is below the usual threshold of 0.05, so the difference is statistically significant.

The 95% [confidence interval](/blog/confidence-interval/) of the difference ranges from 0.15 to 0.89. It shows how imprecise the 0.52 is with so few respondents: Hamburg's true lead may be small or large. Zero, that is, no difference, is not in the interval. The article [The t-test](/blog/the-t-test/) shows how t, the degrees of freedom and the effect size Cohen's d are calculated.

The later examples state these figures in a single line. Other tests report a different test statistic instead of t, for example χ², U or F. Which values indicate a difference depends on the test, and each example explains how to read them. The p-value is always read the same way: it tells you how likely a result at least this extreme would be if there were in fact no difference.

### Proportions: the cross-tabulation with the chi-square test

- **When:** Between groups, you're comparing not a mean but a proportion or a distribution across categories: how many would book again, which format is preferred? Each person appears in the table exactly once.
- **Typical mistake:** Running the test on percentages instead of frequencies, or concluding from a significant result that the relationship is strong. The test doesn't say how strong the relationship is (Schumann 2019, p. 209).

Comparing two proportions and testing the relationship between two categorical variables are the same test, just asked differently. Instead of a separate test for two proportions, you can set up a cross-tabulation with two rows and two columns and run the chi-square test (Malhotra and Birks 2007, p. 534). For this, the expected frequencies have to be large enough (Schumann 2019, p. 206). If they are too small in a 2 × 2 table, use Fisher's exact test. The [calculator in the article on cross-tabulation](/blog/cross-tabulation/#calculator) shows it in that case.

> **Example.** Here the chi-square test checks whether the difference between 75% yes in Hamburg and 55% in Cologne (60 respondents each) still fits chance alone: with p = 0.022, it is significant.
> 
> "Would you book a course with us again?" In Hamburg, 45 of 60 respondents say yes (75%), in Cologne 33 of 60 (55%).
> 
> |  | yes | no | Total |
> |------|------|------|------|
> | Hamburg | 45 | 15 | 60 |
> | Cologne | 33 | 27 | 60 |
> | Total | 78 | 42 | 120 |
> 
> Without a continuity correction, the chi-square test gives χ² = 5.27 with one degree of freedom, p = 0.022, phi = 0.21. The test statistic χ² (chi-square) measures how far the four frequencies in the table are from those you would expect if the proportions were the same in both cities. Some programs run a test for two proportions instead and report z = 2.30. That is the same test in a different notation: z² is exactly χ² (more precisely: 2.2967² = 5.27), and the p-value is the same. The difference of 20 percentage points is significant with this number of responses. Phi indicates how strong it is. By the common rules of thumb (0.1 small, 0.3 medium), phi lies between a small and a medium effect, see [Cramér's V](/blog/cross-tabulation/#cramers-v). For the calculation, see [The chi-square test](/blog/the-chi-square-test/). The article [Cross-tabulation](/blog/cross-tabulation/#percentages) explains which percentages to report.

### Ordered categories: the Mann-Whitney U test

- **When:** Two independent groups, and the variable is ordinal, for example [a single question with five response options](/blog/rating-scale/) that you don't want to treat as metric (Malhotra and Birks 2007, p. 534). It is also the usual alternative to the t-test for small groups with a very skewed distribution (see [The t-test](/blog/the-t-test/#voraussetzungen)).
- **What the result says:** All answers are put into one common ranking. The test checks whether the values of one group tend to lie higher in it than those of the other (Malhotra and Birks 2007, p. 534).
- **Typical mistake:** Reading it as a test of equal medians. In general, it isn't one (Divine et al. 2018, p. 278). So report the distribution of the answers, not just two medians.

> **Example.** Here the Mann-Whitney U test compares 15 answers each from Hamburg and Cologne to a question on a five-point scale: the Hamburg answers lie clearly higher in the common ranking, and the difference is significant (p = 0.014).
> 
> "How satisfied were you with the organization of the course?", a single question on a five-point scale from "very dissatisfied" (1) to "very satisfied" (5), 15 answers per location.
> 
> | Location | 1 | 2 | 3 | 4 | 5 | Median | Mean rank |
> |------|------|------|------|------|------|------|------|
> | Hamburg | 0 | 1 | 3 | 6 | 5 | 4 | 19.3 |
> | Cologne | 1 | 4 | 5 | 4 | 1 | 3 | 11.7 |
> 
> The Mann-Whitney U test gives U = 55, p = 0.014. U comes from comparing every Cologne answer with every Hamburg answer, which makes 15 × 15 = 225 pairs. In 55 of them, Cologne is ahead, with a tie counting as half a pair. Without a difference between the locations, you would expect about half, that is, around 112. The answers from Hamburg lie clearly higher in the common ranking. For comparison: a t-test on the same data would give p = 0.011. Both methods lead to the same conclusion here. That is often the case, but it isn't guaranteed. Decide which one to report before the analysis, not by which p-value turns out smaller.

## Which statistical test for more than two groups?

- **When:** Three or more independent groups, for example three locations or four age groups. For metric variables, one-way analysis of variance (ANOVA) is the standard method, for ordinal ones the Kruskal-Wallis test, an extension of the Mann-Whitney U test (Malhotra and Birks 2007, pp. 546, 565). For proportions and categories, the chi-square test in the cross-tabulation remains the method to use.
- **Assumption:** ANOVA has the same requirements as the independent-samples t-test. The Kruskal-Wallis test needs independent observations and at least ordinal data.
- **What the result says:** Whether the groups differ overall. The overall test doesn't say which group differs from which. For that, there are contrasts planned in advance or pairwise comparisons afterward with a correction (Malhotra and Birks 2007, p. 564).
- **Typical mistake:** Testing all pairs separately with t-tests instead of one overall test and reporting only the significant ones. If all groups were in fact the same and the tests were independent, the probability of at least one false positive would already be 14.3% with three tests instead of 5% (see [Many tests, many false positives](/blog/statistical-significance/#multiple-testing)).

> **Example.** Here three locations with eight answers each are compared (Hamburg 4.09, Cologne 3.56, Munich 4.00): overall, they differ significantly (p = 0.029), but in the pairwise comparison only Hamburg and Cologne do (p = 0.033).
> 
> The satisfaction scale as above, this time three locations with eight answers each: Hamburg 4.09, Cologne 3.56, Munich 4.00. ANOVA gives F(2, 21) = 4.21, p = 0.029. The test statistic F compares how far apart the three location means are with the spread of the answers within the locations. The two numbers in parentheses are again degrees of freedom: three groups minus one, and 24 answers minus three groups. The Kruskal-Wallis test gives H = 6.13, p = 0.047. H is the counterpart of F, based on ranks rather than means. Both say: the three locations differ more than you would expect if the locations were the same. The pairwise comparisons with Tukey's HSD test, which statistics programs offer along with ANOVA, show where this comes from. They calculate the p-values so that the risk of a false positive is at most 5% for all three comparisons together:
> 
> | Comparison | Difference between the means (absolute value) | p by Tukey's HSD |
> |------|------|------|
> | Hamburg and Cologne | 0.53 | 0.033 |
> | Cologne and Munich | 0.44 | 0.088 |
> | Hamburg and Munich | 0.09 | 0.882 |
> 
> *Invented example, my own calculation.*
> 
> After the correction, only the difference between Hamburg and Cologne is statistically significant (p = 0.033). Cologne and Munich differ almost as much in the example (0.44 compared with 0.53 points), but with eight answers per location that isn't enough (p = 0.088). For the report, this means: mean satisfaction is significantly higher in Hamburg than in Cologne. For the difference between Cologne and Munich, the data in this example aren't enough.

## Which statistical test when the same people answer twice?

Ten people go on a diet, and you want to know whether it works. You put all ten on the scale before and after. The people are very different: the lightest weighs 60 kilograms, the heaviest 110. After the diet, each of them weighs one to three kilograms less.

If you compare only the two averages, you see a difference of just over two kilograms. That is small compared with the 50 kilograms between the lightest and the heaviest person. If these were two different groups of ten people each, such a difference could easily come from a few heavier people happening to be in one group. But they are the same people. So you can take a different approach: you compare each person with themselves and note how much weight they have lost. Whether someone weighs 60 or 110 kilograms no longer matters. Each person's own weight is part of both measurements and cancels out when you subtract one from the other. What remains is ten differences, and all of them point in the same direction. This reveals what gets lost when you compare the averages.

Survey data are similar. Respondents are different, too: one person generally rates her skills high, another generally rates his low. If the same people answer twice, for example before and after a course, this personal tendency is part of both answers. A test for paired data therefore works the way you did at the scale: for each person, it takes the difference between their two answers and then tests only these differences. Each person serves as their own comparison. Whatever makes both of their answers higher or lower to the same degree cancels out when you subtract. What remains is how much each person has changed, and that can differ from person to person.

Data are paired when each measurement in one series can be matched to exactly one measurement in the other: the same people before and after a course, or the same people rating two brands or two features of a product (Malhotra and Birks 2007, p. 531). For this, you need a key in both measurements that links the answers of one person.

Two mix-ups are possible:

- **Testing paired data like two separate groups.** The test for independent groups knows nothing about the pairs. It measures the change against the differences between the people, like the two kilograms against the 50 kilograms. Because the same person usually answers the second time much as they did the first, the change then looks less certain than it is, and a real change is more easily missed. The [example below](#paired-t-test) shows this in numbers.
- **Treating separate groups like paired data.** An annual customer survey with newly drawn respondents each time is not a paired sample, even if the topic is "change." There are no pairs here: nobody can be compared with themselves. If you line the rows up as pairs anyway, you are working with a pairing that means nothing. In that case, you compare two independent groups, as in the example [Has satisfaction increased?](/blog/statistical-significance/#worked-example-has-satisfaction-increased).

In short: independent means that the groups contain different people whose answers are unrelated to one another, and the test compares one group with the other. Paired means that the same people were measured twice, and the test compares each person with themselves. If you test paired data like two independent groups, you lose this advantage and are more likely to miss a change.

The following methods apply to exactly two measurements. With three or more measurements of the same people, for example three survey waves or three rows of a matrix question, they aren't enough. For metric variables, there is repeated measures analysis of variance for this (Malhotra and Birks 2007, p. 564). For ordinal data, Malhotra and Birks (2007, p. 338) name Friedman's analysis of variance by ranks without describing it further. You run both with exported data in a statistics program. This article doesn't cover these methods.

### Means: the paired t-test

- **When:** The same people, two measurements, a variable treated as metric.
- **Assumption:** The differences per person are approximately normally distributed, or there are enough pairs. The pairs are independent of each other.
- **What the result says:** The test takes the difference for each person and checks whether the mean of these differences differs from zero (Malhotra and Birks 2007, p. 531).
- **Typical mistake:** Testing the two series of measurements like two independent groups. The example shows what that costs.

> **Example.** Ten participants rate their confidence before and after a course: the paired t-test shows a significant improvement of 0.48 points on average (p < 0.001), while the same change, analyzed like two separate groups, looks less clear-cut (p = 0.019).
> 
> Ten participants rate their confidence in using the program before and after a spreadsheet course (mean of four questions, 1 to 5).
> 
> | Measurement | Mean |
> |------|------|
> | before | 2.90 |
> | after | 3.38 |
> | difference per person, mean | 0.48 (standard deviation 0.28) |
> 
> The paired t-test gives t = 5.46 with 9 degrees of freedom (ten differences, nine of them free), p < 0.001. The 95% confidence interval of the mean difference ranges from 0.28 to 0.67. I come across this mix-up in analyses now and then: before-and-after data are tested like two separate groups. If you analyze the data in this example that way, you get t = 2.58 and p = 0.019. The change itself is the same. It is now just measured against the differences between the ten people, so it looks less clear-cut. The conclusion would be the same here, but the evidence would be much weaker, and with a smaller change the result could flip.

One more point for the report: a before-and-after comparison without a control group shows that something has changed, not why.

### Ordered categories and yes/no: the Wilcoxon signed-rank test and the McNemar test

- **When:** The same people, two measurements, an ordinal variable or doubts about the normal distribution of the differences, provided the assumption named below is met. The Wilcoxon signed-rank test is the counterpart of the paired t-test (Malhotra and Birks 2007, p. 535).
- **What the result says:** The test ranks the differences per person and compares the rank sums of the positive and the negative changes (Malhotra and Birks 2007, p. 535).
- **Assumption:** The test takes into account the size of the differences, not just their direction (Malhotra and Birks 2007, p. 535). So it must be possible to order the absolute differences meaningfully by size. For a single question with ordered response options, this means: you treat a change of two points as larger than a change of one point, no matter where on the scale it happens. If you don't want to assume that, count only the direction of the change (sign test, Malhotra and Birks 2007, p. 536).
- **Typical mistake:** Overlooking that pairs without a change usually drop out of the calculation. With many unchanged answers, the number of cases shrinks.

> **Example.** In the before-and-after example above, nine of ten people improved, one stayed the same, and nobody got worse. The Wilcoxon signed-rank test gives p < 0.01, both with the exact calculation and with the approximation that many programs report.

If the question is about a yes/no proportion in the same people, for example "is aware of the offer" before and after a campaign, the McNemar test is the right method (Malhotra and Birks 2007, p. 536). An ordinary cross-tabulation "before by after" with a chi-square test doesn't answer this question: it tests whether the two measurements are related, not whether the proportion has changed.

## Which statistical test checks a relationship?

With questions about relationships, you don't compare groups. You look at two variables of the same people. Which measure fits depends on the level of measurement of both variables (Schumann 2019, p. 239).

### Two categorical variables: cross-tabulation and the chi-square test

The procedure from the section [Proportions](#proportions) applies here as well, also with more than two categories. The test checks whether there is a relationship, and measures such as phi or Cramér's V show how strong it is (Schumann 2019, pp. 209, 212-213). If one of the variables is ordinal, the chi-square test doesn't use the order. Everything else is in the article [Cross-tabulation](/blog/cross-tabulation/).

### Two ranked variables or two measured quantities: correlation

- **When:** Both variables are numbers or ordered categories, and you're asking "the more of one, the more (or less) of the other?" If both are metric, use the Pearson correlation. If at least one is ordinal, use Spearman's rank correlation. If very many answers fall on the same values, Kendall's tau is considered an alternative (Malhotra and Birks 2007, pp. 575, 581).
- **Assumption:** Pearson measures only linear relationships. Rank correlation works with ranks and also captures relationships that run in one direction without being linear.
- **Typical mistake:** Reading a cause into a correlation, or concluding from a coefficient near zero that there is no relationship. With Pearson, a value near zero can hide nonlinear relationships ([slide 211](/principles-of-survey-research/chapter-5/slide-211.html)).

> **Example.** For twelve customers, we examine whether likelihood to recommend increases with the number of years as a customer: both correlations are 0.79 (p = 0.002), which indicates a strong relationship but doesn't establish causation.
> 
> The data cover twelve customers: their years as a customer and their likelihood to recommend on a scale from 0 to 10.
> 
> | Years as a customer | 1 | 1 | 2 | 2 | 3 | 4 | 5 | 5 | 6 | 8 | 9 | 10 |
> |------|------|------|------|------|------|------|------|------|------|------|------|------|
> | Likelihood to recommend | 5 | 7 | 6 | 4 | 7 | 6 | 8 | 7 | 9 | 7 | 9 | 10 |
> 
> Pearson's r is 0.79 (p = 0.002), and Spearman's rho is also 0.79 (p = 0.002). Both coefficients always lie between −1 and +1. The closer the coefficient is to +1, the more consistently a longer customer relationship goes with a higher likelihood to recommend. Customers who have been with the company longer are more likely to recommend it. The correlation doesn't tell you whether a longer relationship makes people more likely to recommend the company or whether satisfied customers simply stay longer. If the data don't come from a controlled experiment, you can only claim that a relationship exists, not its causal direction ([slide 209](/principles-of-survey-research/chapter-5/slide-209.html)). More on this in the article [Correlation and causation](/blog/correlation-vs-causation/).

### Estimating one quantity from another: regression

Use regression when you want to know by how much a metric quantity changes on average when another one increases, or when you want to take several influencing variables into account at the same time (Malhotra and Birks 2007, pp. 581-582). You get an equation with a slope, its p-value and the proportion of variance explained, R². "Dependent" and "independent" variable describe the calculation, not cause and effect (ibid., p. 581). More on this in the article [Correlation and causation](/blog/correlation-vs-causation/).

## How precise is a single value?

Not every question needs a test. If you want to know how reliable a single percentage is, a confidence interval helps more than any p-value.

> **Example.** 78 of 120 respondents would book again, which is 65%. The 95% confidence interval (normal approximation) ranges from 56.5% to 73.5%, and the margin of error is ±8.5 percentage points. You read the proportion and the number of responses from the report and calculate the interval with the [calculator in the article on the p-value and statistical significance](/blog/statistical-significance/#calculator). The article [Confidence interval](/blog/confidence-interval/) explains how to read a confidence interval correctly.

Statistics programs report the confidence interval for a mean together with the t-test. If you want to test a mean against a fixed target value, for example "Do we reach 4.0 on average?", the one-sample t-test is the right method. You'll find it in QUESTIONSTAR's reports, and it is worked through in the article [The t-test](/blog/the-t-test/#einstichproben).

## What should you check before every statistical test?

First you check whether the requirements are met. Only then do the hypotheses and the calculation follow (Schumann 2019, p. 207).

1. **A random sample?** Significance tests generalize from the sample to a population and, strictly speaking, require random selection (Jacob et al. 2019, p. 71). Without it, the limitation belongs in the report, see [Does this also apply to my online survey?](/blog/statistical-significance/#random-sample).
2. **Enough cases per group or cell?** Jacob et al. (2019, p. 72) recommend at least 20 to 25 cases per subgroup. That is not a general minimum number of cases for all tests. The examples here are deliberately smaller. In cross-tabulations, what counts is the [expected frequencies per cell](/blog/cross-tabulation/#expected-below-5).
3. **Each person only once?** Two measurements of the same person need a method for [paired samples](#paired). For questions that allow several answers, test each answer option as its own yes/no variable.
4. **A planned comparison or one of many?** Decide beforehand which few comparisons you'll test, and report everything else as exploration, see [Many tests, many false positives](/blog/statistical-significance/#multiple-testing).

And after the test: always report the size of the difference as well, because [significant doesn't mean important](/blog/statistical-significance/#practical-significance).

## Which statistical tests does QUESTIONSTAR calculate itself?

You can run the common methods from the overview directly in QUESTIONSTAR's reports, without exporting anything:

- **t-test:** for one sample (a mean against a target value), paired, and for independent samples ([Features](/features/)).
- **Correlations:** Pearson's r, Spearman's rho and Kendall's tau, for a single pair of questions or as a matrix of several questions.
- **Regression analysis:** also with several independent variables, with R² and adjusted R².
- **Cross-tabulations with the chi-square test:** The "Crosstab" report shows chi-square, degrees of freedom and the p-value, plus phi, the contingency coefficient, Cramér's V and lambda. If an expected frequency is below 5, the report points this out.

The reports work within one survey. The paired t-test therefore compares two questions of the same questionnaire, that is, two answers from each person. The reports don't compare different surveys with each other. If before and after are in two separate surveys, you link the answers through a shared key: with an AI assistant via the MCP interface, or by hand in the exported data.

These analyses are included in all plans, including the free one.

If you work with an AI assistant that is connected to QUESTIONSTAR via the [MCP interface](/help/integrations/mcp-ai/), it can start three calculations in QUESTIONSTAR: the paired t-test, correlations and regressions. The calculation runs on QUESTIONSTAR's server. The assistant calls it and explains the result. Via this interface, too, the paired t-test compares exactly two questions of the same survey. The independent-samples t-test is not available as a built-in calculation via this interface, but it is available in the reports.

What the reports don't provide: confidence intervals for proportions, analysis of variance (ANOVA), the rank tests (Mann-Whitney U test, Wilcoxon signed-rank test, Kruskal-Wallis test) and the McNemar test. QUESTIONSTAR doesn't calculate a factor analysis either. For these methods, take one of the two routes in the next section.

**From questionnaire to test in one tool**

Create your survey, collect answers and compare groups in the report: with the t-test, correlation, regression and cross-tabulation with chi-square. All of it in the free plan too.

[Start for free](/order/?code=free)

## How to calculate this with your data

For methods you won't find in the reports, such as ANOVA, the Mann-Whitney U test or the McNemar test, you have two routes.

**With a statistics program.** Export the responses as an SPSS, CSV or XLSX file and run the analysis in SPSS, R or the free program JASP. Check beforehand how opt-out answers such as "can't judge" are coded, so that they don't enter the test as a scale value.

**With an AI assistant.** If QUESTIONSTAR is connected to an AI assistant via the [MCP interface](/help/integrations/mcp-ai/), for example in ChatGPT or Claude, the assistant can read the responses to your survey and calculate the test from them. Here the assistant does the calculation, not QUESTIONSTAR. Name the method explicitly ("Mann-Whitney U test, two-sided"; two-sided means that a difference in either direction is tested, which is the usual case) and have it report the numbers of cases, the test statistic and the p-value. Anything that goes into a thesis, a publication or an important decision should be checked in a statistics program.

Check beforehand which questions the assistant gets to see: free-text answers can contain personal data. The details are under [Rank tests](/blog/levels-of-measurement/#rank-tests) and in the [help article on the MCP interface](/help/integrations/mcp-ai/).

## Checklist: choosing the right statistical test

- The four things are clear: type of question, level of measurement of each variable, number of groups, independent or paired
- Test chosen before looking at the results, not by the smaller p-value
- Checked: random sample or not, number of cases per group, expected frequencies per cell, each person only once
- Number of planned comparisons noted, everything else marked as exploration
- Stated in the report: method, numbers of cases, test statistic (the number the test reports, for example t or χ²) with degrees of freedom where the test reports them, exact p-value, and the size of the difference or relationship

## Frequently asked questions about choosing a statistical test

### Which statistical test should I use for my data?

That depends on four things: the type of your question (difference, relationship, change or precision of a single value), the level of measurement of the variables, the number of groups, and whether the groups are independent or paired (Malhotra and Birks 2007, pp. 493, 525-526). For two means, the independent-samples or the paired t-test fits, depending on the structure of the data. For frequencies in categories with independent groups, it is the chi-square test, and for two measured quantities, correlation.

### When do I use the t-test and when the chi-square test?

Use the t-test when you compare the means of a metric variable between two groups, for example average satisfaction at two locations. Use the chi-square test when both variables are categories and you compare frequencies in a cross-tabulation, for example location and preferred format.

### What is the difference between independent and paired samples?

Independent samples consist of different people whose answers are unrelated to one another. Paired samples contain two measurements of the same people, for example before and after a training course (Malhotra and Birks 2007, p. 525). They have their own methods: the paired t-test, the Wilcoxon signed-rank test and, for yes/no answers, the McNemar test.

### Which statistical test do I use for more than two groups?

With three or more independent groups and a metric variable, one-way analysis of variance (ANOVA) is the standard method, with ordinal data the Kruskal-Wallis test (Malhotra and Birks 2007, pp. 546, 565). For categories, the chi-square test remains the method to use. Several t-tests in a row increase the risk of false positives.

### Which statistical tests can I run in QUESTIONSTAR?

In QUESTIONSTAR's reports, you run the t-test (for one sample, paired and independent), Pearson, Spearman and Kendall correlations, regression analyses and cross-tabulations with the chi-square test. The reports work within one survey. Analysis of variance, rank tests (the Mann-Whitney U test, the Wilcoxon signed-rank test and the Kruskal-Wallis test) and the McNemar test can be run after an export in a statistics program or with an AI assistant via the MCP interface.

## Sources

Divine, G. W., Norton, H. J., Barón, A. E. and Juarez-Colunga, E. (2018). The Wilcoxon-Mann-Whitney Procedure Fails as a Test of Medians. The American Statistician, 72(3), 278-286. DOI 10.1080/00031305.2017.1305291. p. 278.

Jacob, R., Heinz, A. and Décieux, J. P. (2019). Umfrage. Einführung in die Methoden der Umfrageforschung [Surveys: an introduction to the methods of survey research] (4th ed.). Berlin/Boston: De Gruyter Oldenbourg. In German. pp. 71-72.

Malhotra, N. K. and Birks, D. F. (2007). Marketing Research. An Applied Approach (3rd European edition). Harlow: Pearson Education. pp. 338, 492-494, 525-536, 546, 564-565, 575, 581-582.

Marx, P. (2026). Principles of Survey Research. Online course, chapter 5, slides 185, 192, 209 and 211. QUESTIONSTAR. [Slide 185](/principles-of-survey-research/chapter-5/slide-185.html), [slide 192](/principles-of-survey-research/chapter-5/slide-192.html), [slide 209](/principles-of-survey-research/chapter-5/slide-209.html), [slide 211](/principles-of-survey-research/chapter-5/slide-211.html).

Schumann, S. (2019). Repräsentative Umfrage. Praxisorientierte Einführung in empirische Methoden und statistische Analyseverfahren [Representative surveys: a practical introduction to empirical methods and statistical analysis] (7th ed.). Berlin/Boston: De Gruyter Oldenbourg. In German. pp. 206-207, 209, 212-213, 239.

**Ready for your next analysis?**

Create your survey, collect answers, compare groups. Start for free, no credit card required.

[Start for free](/order/?code=free)

I hope your analyses always start with the question and use a test that fits it, not the other way around.

**Related articles:**

- [The t-test: comparing means the methodologically clean way](/blog/the-t-test/)
- [The chi-square test: testing associations between categorical variables](/blog/the-chi-square-test/)
- [P-value and statistical significance, explained simply](/blog/statistical-significance/)

*Dr. Paul Marx*

Author: Dr. Paul Marx
Dr. Paul Marx is the founder and Managing Director of QUESTIONSTAR and was a professor of marketing at the University of Siegen.

## Related articles

- [the-t-test](/blog/the-t-test)
- [the-chi-square-test](/blog/the-chi-square-test)
- [statistical-significance](/blog/statistical-significance)

*Author: Dr. Paul Marx — see [About](/about).*