QUESTIONSTAR
Analysis·November 15, 2026·29 min read

Understanding hypothesis tests: one idea behind z, t, χ² and F

Why do all statistical tests work with a null hypothesis, a test statistic and a value from a table?

For practitioners
Watercolor: four bell curves side by side on one line, each with a small red tail on the right.

Why every test is built the same way: from the bell curve and the standard normal distribution to t, degrees of freedom, chi-square and F. Based on a lecture.

In short: every hypothesis test follows the same idea. You start with your assumption, the alternative hypothesis H1, and set up the null hypothesis H0 to test against it: there is no effect, so no difference and no relationship. Then you ask how likely a value as extreme as yours, or more extreme, would be if H0 were true. This probability is called the p-value. If it isn't larger than the significance level you set beforehand, usually 5%, you reject H0, provided the assumptions of the test are met. That doesn't prove the effect you suspect. The tests differ mainly in their test statistic and its distribution. If you only want to read test results in reports, you'll find the essentials in the section How to read a test result.

This article is based on one of my lectures and also touches on where some of the tests come from. If you know the shared idea, you understand better what a test tells you, and you can use it without simply following a recipe.

Step by step:

  1. What idea underlies every test?
  2. Where does it all begin? The bell curve
  3. Why 5%? A convention from which 1.96 follows
  4. How do you turn all bell curves into one?
  5. What changes when you test a mean?
  6. Does your value lie in the middle or in the tail?
  7. Why do you need t and degrees of freedom?
  8. What are degrees of freedom?
  9. How do you read the table and decide?
  10. t-test, chi-square, analysis of variance, regression: the same idea
  11. What do all tests have in common?
  12. How to read a test result
  13. Frequently asked questions about hypothesis tests
  14. Sources

What idea underlies every test?

A test begins with the hypothesis that an effect exists. What kinds of effects can these be? An ad campaign makes a company earn more or less revenue. Smoking affects whether someone develops lung cancer. Gender influences how people shop. This assumption is called the alternative hypothesis, H1 for short. Opposite it stands the null hypothesis H0: there is no effect. In the last example, that would mean men and women shop the same way. Why test this one in particular? For "no effect," you can calculate exactly which values you'd expect. For an effect whose size is open, you can't. On top of that, refuting is easier than proving. Even a thousand white swans don't prove that all swans are white, but a single black one refutes it. A test works in a similar way, but not in exactly the same way. Data that would be very rare under H0 speak against H0. Unlike the black swan, though, they don't refute H0 with certainty, because rare doesn't mean impossible. They can prove neither that there is no effect nor that there is one.

Then you collect data, calculate test statistics from them and check whether your value is likely under a particular distribution. Every test needs this distribution. It describes which values of the test statistic you'd expect if H0 is true and only chance is at work.

If such an extreme value would be rare there, your observation speaks against H0, and you reject it. How rare it has to be, you set beforehand, with the significance level α, usually 0.05 or smaller. In doing so, you also set the risk of rejecting H0 even though it's true.

For the comparison, you need two numbers. You calculate the observed value of the test statistic from your data. You look up the critical value for your significance level in a table, which is why it's also called the table value, or you let software calculate it. Then you compare the two. For a two-sided z-test or t-test, this means: if the absolute value of the observed value is larger than the positive critical value, your value lies in the rejection region. What the p-value tells you here is explained in the article P-value and statistical significance.

But why a distribution, why a table, why 5% of all things? For that, we have to go back to the beginning.

Where does it all begin? The bell curve

It all starts with an observation: many characteristics are distributed roughly in the shape of a bell. Carl Friedrich Gauss described this bell curve. Later, mathematicians, among them Jarl Waldemar Lindeberg and Paul Lévy, showed why sums and means of many small random influences are, under certain conditions, approximately distributed this way. This statement is called the central limit theorem.

The values are spread around the population mean, written μ (pronounced "mu"). That's where the bell has its peak, and values close to it occur most often. The farther an observation lies to the left or right of it, the less often it occurs. This distribution is called the normal distribution.

What does random mean here? Not that there are no causes. Many small influences act on what you observe, and none of them tips the balance on its own. Then a bell often emerges. It doesn't have to: random values can also be distributed quite differently.

Two examples. The first is women's shoe sizes. Let's assume the most common ones are around EU size 38, 39. Size 42 exists too, but it's much rarer. Size 35 exists as well, but compared to 38 it's rare.

The second is the weight of 500 g packs of coffee as they come off the line. The machine fills them almost exactly. Most packs weigh very close to 500 g, a few weigh a few grams more or less. And the farther from 500 g, the rarer such a pack is. Let's say almost all of them weigh between 495 and 505 g.

Both curves have the same bell shape, but each one sits on its own scale.

Three bell curves side by side: on the left, women's shoe sizes with a peak at about 38 and edges at 35 and 42; in the middle, the weight of 500 g packs of coffee with a peak at 500 g and edges at 495 and 505 g; on the right, in general, any characteristic distributed this way, divided into standard deviations around the mean μ.
Figure 1: The same bell shape on three scales: women's shoe sizes, 500 g packs of coffee and, in general, any characteristic distributed this way (illustrative values).

Why 5%? A convention from which 1.96 follows

The good thing about the normal distribution is that you can say something fixed about it. How widely the values are spread around the mean is measured by the standard deviation σ (sigma). From μ − σ to μ + σ lie about 68% of all values. At exactly two σ, it's about 95.45%. That's why you often hear the simplified version: two σ, 95%.

A two-sided test usually works with the magic 5%. There's nothing magic about them, though. They're a convention: people agreed to treat the outermost 5% of values as rare, 2.5% on the left and 2.5% on the right. The boundary follows from this choice. For exactly 95% to lie in the middle, you have to go 1.96 σ from the mean in each direction, so almost two. Depending on the question, you may be stricter, and then the boundary moves farther out. If an observed value falls into one of these tails, it means: if there were no effect, such a value would be rare. Nothing more. It doesn't tell you how likely an effect is, and it doesn't rule out chance either: even without an effect, a value lands there in 5% of cases.

Bell curve of women's shoe sizes, divided by dashed lines into steps of one standard deviation each, with an arrow marked σ under each step. Below it, three rows: shoe size 36.1 to 40.9, the same positions as μ − 2σ to μ + 2σ and as z-values −2 to +2. Between −1 and +1 about 68%, between −1.96 and +1.96 95%, outside 2.5% on each side.
Figure 2: One step on the z-scale is one standard deviation. z = 1 means one standard deviation above the mean, which for the shoe sizes is size 39.7 (example values). Between −1 and +1 lie about 68%, between −1.96 and +1.96 95%.

This way, every observed value can be checked against its distribution. But each bell sits on its own scale, and you'd need a separate table for every characteristic with its mean and its standard deviation. In practice, that's extremely impractical. And there's a second problem: you usually don't even know the standard deviation of the population. The t-test addresses this problem when testing means. More on that below.

How do you turn all bell curves into one?

The idea is simple. The normal distributions are all built the same way anyway. Only the mean and the standard deviation differ. So you can convert each of them into a single one, the standard normal distribution, with a mean of 0 and a standard deviation of 1. For it, you can determine exactly how likely a value up to a certain point is if only chance is at work. That's in a table, and this one table is enough for every normally distributed quantity whose mean and standard deviation you know.

How do you get from your own data to this curve? Suppose you've measured women's shoe sizes. The horizontal axis shows the possible values, the vertical axis their frequencies. Let's assume your observations form roughly a normal distribution with a mean μ. You transform it into the standard normal distribution in two steps.

First, the mean has to move to zero. To do that, you subtract the mean μ from every value x. Second, the standard deviation has to become one. To do that, you divide the difference between x and μ by the standard deviation σ:

z = \frac{x - \mu}{\sigma}

Here x is a single value, μ the mean of the distribution and σ its standard deviation. The result is called the z-value, and the conversion is called the z-transformation. The z-value tells you how many standard deviations x lies from the mean. If x is normally distributed, you'll find your x on the standard normal distribution exactly at position z.

On the left, a bell curve of women's shoe sizes with mean μ and standard deviation σ; on the right, the standard normal distribution with mean 0 and standard deviation 1; between them, an arrow with the formula z = (x − μ) / σ. On the right, the middle 95% are marked gray and the two tails of 2.5% each red.
Figure 3: The z-transformation brings every normal distribution to a mean of 0 and a standard deviation of 1.

What do you do with this? Let's say your shoe sizes have a mean of μ = 38.5 and a standard deviation of σ = 1.2. You're interested in size 39.1. You convert it to the standard normal distribution: z = (39.1 − 38.5) / 1.2 = 0.50. There, using the cumulative distribution function, you read off how likely it is to measure this value or a smaller one purely by chance: about 69%, so roughly 70%. Because the area to the left of 39.1 under your bell is exactly as large as the area to the left of 0.50 under the standard normal distribution, the 69% applies to size 39.1 as well: about 69% of shoe sizes are smaller. A pretty simple idea, isn't it?

Three parts. Top left, the bell curve of shoe sizes from 35 to 42 with a mean of 38.5, size 39.1 marked in red and the area to its left in teal. A red arrow leads up to the right to the standard normal distribution, where the same value lies at z = 0.50. Below it, the cumulative distribution function from 0 to 100%, with about 69% read off at z = 0.50. On the left, the formula z = (x − μ) / σ and the calculation (39.1 − 38.5) / 1.2 = 0.50.
Figure 4: From shoe size 39.1 via z = 0.50 to the probability: about 69% of values are smaller (example values).

What changes when you test a mean?

You're very rarely interested in the probability of a single observation. Usually it's about the mean: does it equal a given value in the population?

You have a sample. It's summarized by its mean x̄ (pronounced "x bar"): each value multiplied by its frequency, everything added up and divided by the number of values. x̄ estimates the population mean μ. It doesn't have to be the most common value, and it doesn't describe the whole distribution either. For the test, something else counts: if a mean like yours would be rare if the null hypothesis were true, that speaks against it.

The classic example: is the mean of women's shoe sizes really 38? This time you standardize the mean x̄. If 38 is the true mean and you subtract 38, you should land at zero or close to it. If you land somewhere off to the side, you check whether such a distance is still quite possible under H0. In the general case, instead of 38 you have μ0 (pronounced "mu zero"), the value you assume the mean to be.

Now it's no longer about the distribution of individual shoe sizes, but about the distribution of the mean. If you drew many samples, you'd get many means, and they form a bell of their own. This is where the central limit theorem comes in: for independent observations from the same distribution with finite variance, the mean of large samples is approximately normally distributed, even if the individual values aren't.

But a mean varies less than a single value. If the observations are independent, its standard deviation is the standard deviation of the population divided by the square root of n, the number of points that formed the mean:

z = \frac{\bar{x} - \mu_0}{\sqrt{\sigma^2 / n}}

Here x̄ is the sample mean, μ0 the value under the null hypothesis (38 in the example), σ2 the variance, that is, the square of the standard deviation, and n the number of observations. This calculation is exact when the observations are independent and normally distributed. With large samples, it often works well enough as an approximation.

Does your value lie in the middle or in the tail?

So if H0 is true, this z-value follows the standard normal distribution, exactly for normally distributed observations, approximately otherwise. Let's see where it lands. If it lands at zero, the probability of measuring this value or a smaller one is 50%, and the complementary probability is also 50%. You can't reject the null hypothesis, because the result is quite possible under H0. You simply lack a reason to reject it.

Up to one standard deviation, you're within the 68%, up to 1.96 standard deviations within the 95%. If H0 is true, you get a value this far from zero or farther with a probability of more than 5%.

Only in the tail, in the 2.5% on the right or left, does the picture change. Take a z-value of 2.1. It's just as far from the mean as −2.1, and because the bell is symmetric, deviating this far or farther to the right is just as likely as to the left. Taking both sides together, if H0 is true, you get a value this far away or farther with a probability of only about 3.6%, so less than 5%. Your value thus falls into the rejection region, and you reject H0, in the example the hypothesis that the true mean of women's shoe sizes is 38. That's how the basic idea of the z-test works.

Two standard normal distributions side by side, the tails beyond −1.96 and 1.96 in red. On the left, our value z = 1.2 lies in the middle 95%, H0 is not rejected. On the right, our value z = 2.1 lies in the right tail, H0 is rejected.
Figure 5: Within the middle 95%, H0 is not rejected, but that doesn't prove it. At z = 2.1, the value lies in the tail, and H0 is rejected.

The confidence interval rests on the same 95%. You already know the standard deviation of the mean, σ/√n, from above. It's called the standard error. Here, the 95% confidence interval extends 1.96 standard errors on either side of the sample mean. More on this in the article Confidence interval: how to calculate and interpret it.

Why do you need t and degrees of freedom?

The z-test has one small detail, but an important one. The σ in the formula stands for the standard deviation of the population. And for most random variables, that's devilishly hard to know. You measure a limited number of points and estimate σ from the standard deviation of the sample, s for short, that is, from how your measured points are spread.

If you put s in place of σ, the value no longer follows the standard normal distribution exactly. That's why it gets a name of its own: t. Its distribution was studied by William Sealy Gosset. He published under the pseudonym Student, hence the name Student's t-distribution.

t = \frac{\bar{x} - \mu_0}{\sqrt{s^2 / n}}

The letters are the same as for z, only s2, the variance of the sample, takes the place of σ2. It's calculated with n − 1 in the denominator. You'll see why when we look at degrees of freedom in a moment. The t-distribution is exact when the observations are independent and normally distributed.

The t-value also follows a bell curve, but it's wider, because estimating σ adds uncertainty. Larger deviations and extreme values become more likely. The more observations you have, the more precise your estimate and the narrower the distribution. As n goes to infinity, it gets closer and closer to the standard normal distribution.

The standard normal distribution as a dark line, with three t-distributions with 2, 5 and 30 degrees of freedom over it. The one with 2 degrees of freedom is the flattest and has the heaviest tails, the one with 30 lies almost on top of the normal distribution.
Figure 6: The t-distribution is flatter and has heavier tails. The more degrees of freedom, the closer it lies to the normal distribution.

That has to do with the degrees of freedom.

What are degrees of freedom?

Look at the mean once more. Suppose you have three observations:

\bar{x} = \frac{x_1 + x_2 + x_3}{3}

The mean is 10. How many of x1, x2 and x3 can be completely free if the result is to be 10? It turns out: only two out of three.

x1 can be any number, for example 5. x2 can also be any number, let's say 15. But then x3 can no longer vary freely. For the result to be 10, it has to be exactly 10: 30 divided by 3 equals 10. With any other number, the equation no longer holds.

Three boxes side by side, above them the formula x̄ = (x₁ + x₂ + x₃) / 3 = 10. The first contains 5, the second 15, both marked as freely chosen. The third has a red border and contains 10, with the note: fixed.
Figure 7: Three values, mean 10. The first two are free, the third then has to be 10: df = 3 − 1 = 2.

So only 3 minus 1 values can vary freely. With four values, it would be three, with five, four, and so on. That's why the one-sample t-test uses:

df = n - 1

df stands for the degrees of freedom, n for the number of observations. As n grows, df grows, and s estimates the true spread σ more and more precisely. With many degrees of freedom, the t-distribution therefore approaches the normal distribution. For the paired t-test, which you'll meet below, it's the same, but there n counts the pairs. For two independent groups, it's n1 + n2 − 2, at least in the classic form, which assumes equally large spread in both groups. The Welch t-test, which many recommend today, calculates the degrees of freedom differently.

How do you read the table and decide?

Because the t-distribution depends on the degrees of freedom, the t-table has two dimensions. The columns show the probability, usually as an error probability, that is, as the significance level. The rows show the number of degrees of freedom: 1 for two data points, 2 for three, and so on.

You set the significance level before the analysis, for example 0.05. Now you've calculated your t-value, and you determine the degrees of freedom that led to it. Then you look for the cell where the column for 5% and the row for your degrees of freedom cross. That's where the critical value is.

An example with numbers. You have nine measurements, so 8 degrees of freedom. For 8 degrees of freedom, the table shows the value 2.306 for 5%, two-sided, rounded 2.3. Pay attention to the column headings: many tables have separate columns for the one-sided and the two-sided test. One-sided, the critical value for 5% at 8 degrees of freedom would be 1.860. Here we calculate two-sided. Your observed t-value is, let's say, 2.4.

On the curve, this means: for the standard normal distribution, the rejection region for 5%, two-sided, begins at ±1.96. Because the t-distribution with 8 degrees of freedom is wider, it begins farther out there, at ±2.306. Your value of 2.4 lies above that, so in the rejection region. The p-value is smaller than 0.05, and you reject H0. That's the basic idea behind the critical value. But there's one thing the p-value doesn't tell you here: how likely it is that your decision is wrong. It tells you how rare a value this extreme or more extreme would be if H0 were true.

On the left, an excerpt from a t-table with rows for degrees of freedom and columns for the significance level, the cell for 8 degrees of freedom and 5% two-sided with the value 2.306 highlighted, next to it in the same row 2.896 for 2%. On the right, a t-distribution with 8 degrees of freedom, the critical value 2.306 as the boundary of the red rejection region and the observed value 2.4 to its right.
Figure 8: With 8 degrees of freedom, tobs = 2.4 lies to the right of tcrit = 2.306, in the rejection region. H0 is rejected.

It also works the other way around. In the row for your degrees of freedom, you compare the values with your observed value until one exceeds it. At the last value that's still smaller, you draw the line up and read off which significance level it corresponds to. In the example, that's 2.306 at 5%. The next value, 2.896 at 2%, is already larger than 2.4. So your value is significant at the 5% level, but not at the 2% level. The p-value lies between 0.02 and 0.05. Software gives the exact value. Here it's 0.043, so indeed a value in this range. How to read it is shown in the section How to read a test result.

t-test, chi-square, analysis of variance, regression: the same idea

The t-test in three variants

There are three common variants of the t-test. You've just seen the first one: it tests whether a mean equals a particular number, for example whether the mean women's shoe size in the population is 38. Often you also test whether a mean in the population differs from zero. Then μ0 = 0, and the calculation becomes simpler: t = x̄ / √(s2/n). If this value falls into the rejection region, the data speak against a mean of zero. Whether the influence you suspect is also the cause, the test alone doesn't tell you. That depends on how your study is designed.

Two groups. The second variant, the two-sample t-test, uses two independent samples to test whether the means of the two populations differ. It's often explained with men and women: do they behave the same way with regard to some phenomenon or not? Do they buy different amounts of, say, sausages, bread rolls or pears? In opinion research, this is how you compare the opinions of different groups. The null hypothesis is μ1 = μ2: the means are equal in the population, and the difference between your sample means x̄1 and x̄2 arose by chance.

Two overlapping bell curves, black with the mean x̄₁ and red with the mean x̄₂, with a double arrow between them for the distance. Title: Two groups: is the distance between the means large compared to the spread?
Figure 9: Two groups, two means. The t-test checks how rare such a large difference would be if there were no difference.

The scheme stays the same. The numerator now holds the difference between the two means. If you want to test whether it equals a number W0 set in advance, you also subtract W0: x̄1 − x̄2 − W0. The denominator holds the estimated standard error of the difference, that is, how much the difference varies by chance. As with z, it takes care of the standardization, only this time two independent samples come together, each with its own size and its own spread. The classic formula assumes that the variance is the same in both populations and combines the two variances, weighted by their degrees of freedom, into a common one. Many experts today recommend going straight to the Welch t-test, because it doesn't require equal spread in both groups. But the pooled formula shows most simply how the test is built. You don't need to memorize the formula:

s_p^2 = \frac{(n_1 - 1)\,s_1^2 + (n_2 - 1)\,s_2^2}{n_1 + n_2 - 2}
t = \frac{\bar{x}_1 - \bar{x}_2}{\sqrt{s_p^2 \left(\frac{1}{n_1} + \frac{1}{n_2}\right)}}

Here x̄1 and x̄2 are the group means, n1 and n2 their sizes, s12 and s22 their variances, and sp2 the common variance of both groups.

Before and after. The third variant is the paired t-test. Here you measure the same people before and after. A typical example: are test subjects different after a treatment in an experiment than before? Did a remedy help, measured by well-being? The calculation uses the difference between the two measurements. It's standardized as in the first variant, because again there's only one sample:

t = \frac{\bar{d} - W_0}{\sqrt{s_d^2 / n}}

Here d̄ is the mean of the differences, W0 the difference you assume, sd2 the variance of the differences and n the number of people, that is, of pairs. Usually W0 = 0. Then H0 is: in the population, there is no difference on average. And even if you reject H0: without a comparison group, it isn't certain that the remedy was the cause. Perhaps time simply did its work. Or conspicuously poor values moved toward the middle on their own at the second measurement. This is called regression to the mean.

So much for the t-test. And the rule of thumb "two σ, 95%"? For the two-sided z-test with α = 0.05, the boundaries are at ±1.96. For the t-test, they depend on the degrees of freedom and, with few degrees of freedom, lie considerably farther out, for example at ±2.306 with 8 degrees of freedom. How to run the t-test with your own data is shown in the article The t-test.

The chi-square test

The chi-square test examines nominal data, that is, data that don't form a ranking but only name things: colors such as red, blue and green, gender, car brands such as Mercedes-Benz, Audi or Volkswagen, models of watches or airplanes, types of companies.

It comes in two forms that use the same formula. The goodness-of-fit test checks whether an observed distribution matches an expected one. For example, you expect a certain distribution of frequencies for three products A, B and C and observe a slightly different one. Is the difference still chance? The test of independence checks whether two characteristics are related, and that's the one our example is about.

For this, you work with a contingency table, or crosstab. The rows show, for example, men and women, the columns whether they buy much or little. Each cell holds the observed value and the expected value, calculated on the assumption that the two characteristics are independent. Let's say you observed 30 men who buy much and 20 who buy little, plus 10 women who buy much and 40 who buy little. Without a relationship, you'd expect 20 and 30 for men and women alike. How to form the expected values is shown in the article on cross-tabulation.

The test statistic is the sum of the differences between the target value, that is, the expected value, and the actual value. So that the differences don't cancel each other out, each one is squared. To normalize it, it's divided by the expected value. You do this for every cell and add everything up:

\chi^2 = \sum \frac{(E - O)^2}{E}

Here O is the observed frequency of a cell and E the frequency expected under H0, which for a crosstab is the one you'd expect if the characteristics are independent. Σ means: add up over all cells. The χ2 distribution applies here only approximately, and even that only with independent observations and expected frequencies that aren't too small. For sparsely populated tables, exact methods are therefore used.

What's new is the shape of the distribution. Because you work with squared values, the negative range isn't accessible for chi-square. The distribution starts at zero and is skewed. As with t, its shape depends on the degrees of freedom, for a crosstab (rows − 1) · (columns − 1). With more degrees of freedom, the curve becomes less skewed and its peak moves to the right. Otherwise, it's the same idea as everywhere.

Let's finish the example. Men who buy much: (20 − 30)² / 20 = 5. Men who buy little: (30 − 20)² / 30 ≈ 3.33. Women who buy much: (20 − 10)² / 20 = 5. Women who buy little: (30 − 40)² / 30 ≈ 3.33. Together:

\chi^2_{obs} = 5 + 3.33 + 5 + 3.33 \approx 16.7

The table has two rows and two columns, so df = (2 − 1) · (2 − 1) = 1. The critical value for 5% at one degree of freedom is 3.84. Your value of 16.7 lies far above that, and you reject H0, that is, the assumption that gender and the amount purchased are unrelated. More in the article The chi-square test.

Top left, pairs of bars for each cell of the crosstab, below them the crosstab itself, with arrows leading from each pair of bars to its cell: men buy much, observed 30, expected 20. Men buy little, observed 20, expected 30. Women buy much, observed 10, expected 20. Women buy little, observed 40, expected 30. Observed as black bars, expected without a relationship as bars with a red outline. On the right, the skewed chi-square distribution, which starts at zero, for 1, 3 and 6 degrees of freedom.
Figure 10: Chi-square compares what was observed with what was expected, cell by cell. Its distribution has no negative range and is skewed.

Analysis of variance

In an analysis of variance, you observe a phenomenon in several groups, let's say in four, and ask whether their means differ significantly. The null hypothesis is:

H_0\!: \mu_1 = \mu_2 = \mu_3 = \mu_4

The t-test compares only two groups. Here the question is whether all of them are equal overall. The alternative hypothesis H1 is: not all of them are equal. For that, it's enough for a single group to step out of line. Whether your data show this is decided by the F-test.

First, you form the mean in each group. Then you take all the data together, without dividing them into groups, and calculate the overall mean. That gives two kinds of deviation: how far each observation deviates from the mean of its group, and how far the same points deviate from the overall mean.

Data points in four groups side by side, title: 1. The measurements in four groups.
Figure 11: Four groups of data points.
The same points with a red horizontal line for the overall mean. Red segments lead from it to every single data point. Title: 2. Overall mean and total variation.
Figure 12: The red line is the overall mean. The red deviations of each point from it make up the total variation.
The same points. Each group has a teal horizontal line for its own mean, from which teal segments lead to the points of the group. The red line of the overall mean remains. Title: 3. Group means and the variation within.
Figure 13: In teal, the group means and the variation within the groups. In red, the overall mean.

What you want to know: is the variation within the groups clearly smaller than the total variation? Then it makes sense to separate the data into the groups. And then there's a lot to suggest that the characteristic you used to divide them is related to the measured value. Whether it's also the cause, the test alone doesn't show. That depends on the design of the study.

For the test statistic, you decompose the total variation. Figure 14 shows all three deviations together. For each point: red = black + teal. Red leads from the overall mean to the point. Black leads from the overall mean to the mean of its group, which is the difference between the groups. Teal leads from the group mean to the point, which is the deviation within the group. Upward counts as plus, downward as minus. That's why red can also be shorter than black, for example when the group lies above the overall mean but the point lies below its group.

The same four groups of data points with the title 4. All together: total = between + within. At each point, a red segment from the overall mean to the point and a teal segment from the group mean to the point. Next to each group, a black double arrow between the group mean and the overall mean. Legend: red total deviation, teal within the group, black between the groups. The segments show directed deviations with signs.
Figure 14: All together. The red total deviation of each point is the sum of the black piece between the groups and the teal piece within the group, each with its sign.

Variation here means the sum of squared deviations, or sum of squares for short. And for these sums of squares, added up over all points, the same decomposition holds:

\text{total variation} = \text{between-group variation} + \text{within-group variation}

The between-group variation is the part that the division into groups explains. The within-group variation is what's left over. F puts these two in relation, each divided by its degrees of freedom:

F = \frac{\text{between-group variation} / df_1}{\text{within-group variation} / df_2}

df1 is the number of groups minus 1, df2 the number of all observations minus the number of groups. If the teal segments are much shorter than the red ones, the between-group variation is large relative to the within-group variation. If the ratio stays large even after both have been divided by their degrees of freedom, F becomes large. The classic F-test assumes that the observations are independent and that, in each group, they are approximately normally distributed with equal spread.

Because the F-distribution has two degrees-of-freedom parameters, its table is organized differently: one direction shows the degrees of freedom of the numerator, the other those of the denominator, and there's a separate F-table for each significance level. The idea is absolutely the same, only the test statistic is a little different.

There's a second step. If the F-test is significant, all you know is that not all means are equal. Which groups differ is shown by follow-up comparisons, so-called post hoc tests. Because you test several pairs at once, you have to correct for multiple testing. Otherwise you'll too often find differences that are just chance.

Regression

In regression analysis, you put a line through a cloud of points. Here we stay with a straight line, that is, with one explanatory variable. With several, the test works the same way. The line is placed so that the squared vertical distances of the points from it are, taken together, as small as possible, following the method of least squares. Here, too, the deviation decomposes. The total deviation runs from the overall mean to the data point. The explained deviation is the piece from the overall mean to the line. The unexplained one runs from the line to the point. Added up with their signs, the two make up the total deviation.

Scatter plot with data points, a red horizontal line for the overall mean and a teal regression line. Title: Regression: total = explained + unexplained. Next to each point, three vertical segments: red from the overall mean to the point (total deviation), black from the overall mean to the line (explained deviation), teal from the line to the point (unexplained deviation). The segments show directed deviations with signs.
Figure 15: In red, the total deviation from the overall mean to the point. In black, the part up to the teal line, which the regression explains. In teal, the rest from the line to the point, which remains unexplained.

To test the regression model, you ask: does the line explain so much more deviation than the mean alone that such an improvement would be rare if H0 were true? For this, you again form an F, the explained sum of squares divided by the unexplained one, both divided by their degrees of freedom:

F = \frac{\text{explained sum of squares} / df_1}{\text{unexplained sum of squares} / df_2}

df1 is the number of explanatory variables, the so-called regressors, df2 the number of observations minus the number of regressors minus 1. The null hypothesis is: all slopes are zero, the line explains nothing. With the F-table, as with the analysis of variance, you determine whether F is too large for that. If so, the regression has achieved something: it explains more than the mean alone. Whether it also predicts new data well has to be checked separately, though. If not, your data aren't sufficient to reject H0 at your significance level. But it doesn't follow that the line explains nothing. This F-test has assumptions, too. They concern the distances of the points from the true line of the population, the so-called errors: they should be independent and normally distributed, average zero and have equal spread everywhere.

You test the individual regressors with t-tests of their own, one per coefficient. The suspected effect lies in their slope, and the t-test asks whether it differs from zero. These t-tests answer a different question than the F-test and don't require it to be significant.

What do all tests have in common?

Behind all four tests stands the same question: how well can what we see in the data be explained by chance alone? It's always answered in the same steps.

First, all tests set up their hypotheses the same way. What's tested isn't your assumption H1, but its opposite, the null hypothesis H0. Why, we saw at the beginning: for "no effect," you can calculate which values chance would lead you to expect. And rare results speak against H0, like the black swan against the white ones, only without final certainty.

Second, all tests calculate a value from the data, the test statistic: z, t, χ2 or F. It measures how far the data deviate from what H0 leads you to expect, relative to how much deviation chance alone produces.

Third, all of them compare this value with a critical value. You don't have to invent it. It follows from the distribution that the test statistic has if H0 is true, and from the significance level you chose beforehand. Each test statistic has its own distribution, and for t, χ2 and F, degrees of freedom come into play. That's why there are four tables, but only one way to read them. By the way, all four distributions can be derived from the standard normal distribution that this article began with.

Fourth, the decision follows. If your value lies beyond the critical value, you reject H0. Otherwise you do not reject H0, without having proven it either.

Side by side, you can see it best:

One idea, four tests

TestWhat is compared?Test statisticDistribution if H0 is trueDegrees of freedomExample in this article
Testz-testa mean with an assumed value, σ knownz = distance / standard errorstandard normal distributionnonewomen's shoes: is the mean 38?
Testt-testa mean with an assumed value, or two means, σ estimatedt = distance / estimated standard errort-distributionn − 1, for two groups n1 + n2 − 2 (Welch: adjusted)t = 2.4 with 8 degrees of freedom, p = 0.043
TestChi-square testobserved with expected frequenciesχ2 = sum of the squared distances, each divided by the expected valueχ2 distributionfor the test of independence (rows − 1) · (columns − 1)men and women, much and little: χ2 ≈ 16.7 with 1 degree of freedom
TestAnalysis of variance (F)between-group variation with within-group variationF = ratio of the two, each divided by its degrees of freedomF-distributiongroups − 1 and observations − groupsfour groups
TestRegression (F)explained with unexplained variationF = ratio of the two, each divided by its degrees of freedomF-distributionexplanatory variables and observations − explanatory variables − 1regression line through the cloud of points

For your next analysis, this means: whether the report says t, χ2 or F, the question behind it is always the same. How likely would such an extreme value be if there were no effect? Which test fits your data is covered in the guide Which statistical test should I use?.

How to read a test result

In practice, a program does the calculation, for example SPSS or R. There you'll usually find the p-value in the column "Sig." or "p". For the t-test, newer SPSS versions show a one-sided and a two-sided p ("One-Sided p" and "Two-Sided p"). If you didn't set a direction beforehand, read the two-sided one. For the t-test with two groups, SPSS shows two rows: "Equal variances assumed" and "Equal variances not assumed". Read the row "Equal variances not assumed", which corresponds to the Welch t-test.

How to write it up. The result includes the test statistic and the p-value, and for t, χ2 and F also the degrees of freedom. For the example from this article, 8 degrees of freedom and tobs = 2.4, that means: "t(8) = 2.40, p = 0.043". Always give the means as well, so x̄ and μ0 when comparing with a given value, and both group means for two groups. Only they show in which direction and how far apart the values are.

Significant doesn't mean important. With a very large number of respondents, even a tiny difference can become significant, for example 0.1 points on a 5-point scale, when two groups of 1,000 respondents each are compared and the answers vary by about one point. So always look at the size of the difference too, and ask whether it matters for your decision. There are dedicated measures for this, the effect sizes. They describe how large a difference is, regardless of how many people were surveyed.

Not significant doesn't mean "no difference." It means: the data aren't sufficient to reject H0. Perhaps there were too few respondents, perhaps the difference is small or the spread large.

Many tests, many chance hits. At 5%, any test can turn out "significant" even without an effect. If you test 40 tables, you have to expect two "significant" results on average, even if there's no effect anywhere.

p doesn't tell you who responded. The p-value captures only random fluctuation, not distortions that arise because respondents selected themselves. Whether you may draw conclusions about everyone from the result depends on who responded. If, for example, only some of the customers you contacted respond, check whether those who responded might differ from the others, for example in age, in the length of the customer relationship or in revenue. What you know about everyone you contacted, you can compare with those who responded. If you find a difference, mention it in the report and read the result mainly as a statement about those who responded.

And once more, the most important point: the p-value tells you how rare a value this extreme or more extreme would be if H0 were true. It doesn't tell you how likely H0 or your effect is. More on this in the article P-value and statistical significance.

Frequently asked questions about hypothesis tests

What is a hypothesis test, explained simply?

A hypothesis test checks whether a result from your data can still be explained by chance. You compare your test statistic with the critical value of a known distribution. If your value lies beyond it, you reject H0, the hypothesis that there is no effect. The data then speak against H0.

Why do you test the null hypothesis and not your own hypothesis?

Because H0 specifies an exact case, usually "no effect." For this case, you can calculate which values you'd expect. For "there is some effect," you can't, because it's open how large it is. If your value lies in the outermost tail of the distribution under H0, that speaks against H0 and for your effect.

Why is the significance level usually 5%?

In a normal distribution, 95% of all values lie within 1.96 standard deviations around the mean, so just under two. In the two-sided z-test with α = 0.05, the remaining 5%, 2.5% each on the left and right, form the rejection region. Even if H0 is true, the test statistic lands there in 5% of cases. The 5% is a convention, not a law of nature.

What is the difference between the z-test and the t-test?

The z-test requires that you know the standard deviation of the population. That's rare. The t-test uses the standard deviation of the sample. Its t-distribution is wider with few degrees of freedom and approaches the normal distribution with many.

Sources

Fischer, H. (2011). A History of the Central Limit Theorem. From Classical to Modern Probability Theory. New York: Springer. DOI 10.1007/978-0-387-87857-7. History of the central limit theorem from Laplace to Lindeberg and Lévy.

NIST/SEMATECH (n.d.). e-Handbook of Statistical Methods, section 1.3.5.3 "Two-Sample t-Test for Equal Means". National Institute of Standards and Technology, itl.nist.gov, accessed November 15, 2026. Covers the pooled variance and the Welch degrees of freedom for unequal variances.

Student (1908). The Probable Error of a Mean. Biometrika, 6(1), 1-25. DOI 10.2307/2331554. The first publication of the t-distribution, under the pseudonym of William Sealy Gosset.

Related articles:

Dr. Paul Marx

Author: Dr. Paul Marx
Dr. Paul Marx is the founder and Managing Director of QUESTIONSTAR and was a professor of marketing at the University of Siegen.