QUESTIONSTAR
Analysis·November 15, 2026·31 min read

One-way analysis of variance: do the groups really differ?

How do you test whether the means of several groups really differ or only by chance?

For practitioners
Watercolor: three clusters of colored dots along one line, each with a small red dash as its mean.

How analysis of variance decomposes variation: sums of squares, degrees of freedom, the F-test and eta squared, step by step with a worked example. Based on a lecture.

In short: one-way analysis of variance tests whether the mean of a measured value differs between groups formed by one characteristic. For example: does satisfaction with customer service differ by contact channel, that is, between phone, chat and email? To find out, it splits the total variation in the responses into two parts: the part that lies between the groups and the rest, within the groups. Each part is divided by its so-called degrees of freedom to make the two quantities comparable. Their ratio gives the F-value. If it is larger than a threshold from the F-table, the data speak against the assumption that all groups are equal on average. Eta squared (η2) measures the strength of the relationship: the share of the variation that the groups explain. In the worked example of this article, F = 10.0 against a threshold of 3.89, and the channel explains 62.5% of the variation.

This article is based on one of my lectures. It follows the lecture step by step, from the idea through the decomposition of variation to the F-test and the strength of the relationship. Every calculation step is written out, so you can follow it with pen and paper.

Step by step:

  1. What does analysis of variance do?
  2. Where is analysis of variance used?
  3. What are a factor and factor levels?
  4. What are the steps of analysis of variance?
  5. Does satisfaction differ by contact channel?
  6. How do you estimate a value without and with knowledge of the group?
  7. How do you decompose the variation?
  8. What does this look like as a model?
  9. How do you calculate the sums of squares?
  10. Why divide by the degrees of freedom?
  11. How does the F-test check whether the difference is significant?
  12. How strong is the relationship?
  13. What does analysis of variance assume?
  14. Which channels differ?
  15. What if several factors are at work?
  16. How to report the result
  17. What to ask when others do the calculations for you
  18. Frequently asked questions about analysis of variance
  19. Sources

What does analysis of variance do?

Analysis of variance deals with two kinds of variables. One is a categorical variable that defines the groups and is treated as the independent variable, the other is measured on a metric scale and is treated as the dependent variable. There can also be more than one of each kind, we'll come to that shortly.

Often the grouping variable is nominal. Nominal means: the values only indicate which group a case belongs to. They have no real numerical meaning. Examples are membership in a group, a color, a brand or gender. Even if you number the groups 1, 2 and 3, you can't calculate with these numbers. Group 2 isn't twice as much as group 1. The groups may also have an order, such as the discount levels in the electronics chain example below. What matters is only that they form groups. Metric, on the other hand, means: the distances between the numbers mean something, you can meaningfully add them up and form means, for example with revenue, visitor numbers or kilograms.

Analysis of variance does two things. First, it tests whether the population means differ between the groups. Second, it measures how strongly the groups and the measured value are related. You always look at this second number, no matter how the test turns out.

That's why it is one of the most common methods in experiments. In an experiment, you divide the participants into groups according to certain criteria. The group someone belongs to is a nominal variable. What you then measure is a metric result, often called the outcome. Analysis of variance tests whether this outcome differs between the groups.

Where is analysis of variance used?

A cinema operator wants to know how well different kinds of advertising for the films he shows work: posters, billboards, newspaper ads, flyers or leaflets in the mailbox. He picks comparable cinemas from his chain, for example in different cities, and uses each type of advertising in several of them. Which cinema gets which advertising is best left to chance, for example by drawing lots. At the end, he compares the visitor numbers. With only one cinema per type of advertising, this wouldn't work: within a group, there would then be no variation to compare the difference between the groups with. Of course, absolutely identical cinemas don't exist in practice. Even comparable cinemas still differ, for example in their audience. Random assignment ensures that such differences are distributed equally across the types of advertising on average. In a single trial, imbalances can still remain. A clear difference in visitor numbers then speaks for an effect of the advertising.

An electronics chain asks whether a price discount of 10 or 20 percent clearly increases sales of a product. It picks comparable stores, grants different discounts in them and observes sales.

In advertising, analysis of variance fits A/B tests with more than two variants, also called A/B/C tests, when a value is measured. You design different ads. The type of ad is a nominal characteristic. What you measure is, for example, how much people like the ad, what feelings it evokes or how much it sells. Many A/B tests, however, only count yes or no, for example whether someone clicks. For that, you use the chi-square test.

In medicine, this is how you test whether drugs work. One group gets the drug, a control group gets a placebo, and at the end you compare treatment outcomes. In the same way, you can compare how different diets affect weight.

But companies don't only need analysis of variance for questions about their customers, they also need it internally. Does employee satisfaction differ between locations or departments? Are three training formats received differently? Here, too, there is a group on one side and a measured value on the other.

What are a factor and factor levels?

Analysis of variance assumes a functional relationship: the model expresses the metric dependent variable Y as a function of one or more categorical independent variables X, plus a random error.

Y = f(X_1, X_2, \ldots, X_J) + \varepsilon

Here, Y is the dependent variable, for example the number of visitors, and X1 to XJ are the independent variables, for example the type of advertising. ε stands for the random deviation: individual values deviate from the mean of their group. In analysis of variance, the X are called factors. Their values are called factor levels. In the cinema example, the type of advertising is the factor, and poster, newspaper ad and flyer are its levels.

Analysis of variance is usually abbreviated ANOVA. Depending on the number of variables, there are several forms:

  • One-way analysis of variance: one dependent variable and one factor. It doesn't get any simpler than this. It is also called single-factor analysis of variance. It is the subject of this article.
  • Factorial analysis of variance: one dependent variable and several factors that are analyzed together. A cinema's visitor numbers may be influenced not only by the type of advertising, but also by its location, in the center or on the outskirts. Location, too, only forms groups, so it is a factor just like the advertising.
  • Multivariate analysis of variance: more than one dependent variable. It is abbreviated MANOVA.

What are the steps of analysis of variance?

Three steps matter to us. First, we formulate the model. Then we decompose the variation. And finally, we test for statistical significance and measure the strength of the relationship. We'll now go through all three with an example.

Does satisfaction differ by contact channel?

A company offers its customers three ways to reach customer service: phone, chat and email. The head of customer service suspects that satisfaction differs by channel, but doesn't know. So after every contact, she asks for a rating: "How satisfied were you with our service?", on a scale from 0 (not at all satisfied) to 10 (completely satisfied).

This gives us one factor, the contact channel, with three factor levels: level 1 phone, level 2 chat, level 3 email. The dependent variable is satisfaction. Strictly speaking, a scale from 0 to 10 only shows an order, but in practice it is treated like measured values, because the distances between the points are taken to be equal (when that is justifiable).

Do we have to make the values comparable first? Not here. Each response is one person's rating on the same scale, no matter whether ten or a thousand people come through the chat. It would be different if we were, say, counting complaints per channel. Perhaps far more contacts come in by phone than by email, and for that reason alone there would be more complaints there. Then we would have to normalize the numbers, for example to complaints per 1,000 contacts, so that we can compare them.

So that every step can be calculated in your head, we take only five responses per channel. The numbers are made up: they are practice numbers, not the result of a study.

Satisfaction by contact channel (made-up practice numbers, scale 0 to 10)

Response kPhone (g = 1)Chat (g = 2)Email (g = 3)
Response k1975
Response k2856
Response k31083
Response k4864
Response k51097
Response kMean975

The question is: does mean satisfaction differ between the contact channels? Our assumption is the alternative hypothesis H1: not all three population means are equal. But, as with every statistical test, what is tested is its opposite, the null hypothesis H0: all three population means are equal. As a formula:

H_0\!: \mu_{\text{Phone}} = \mu_{\text{Chat}} = \mu_{\text{Email}}

Here, μ (pronounced: mu) is the mean satisfaction of all customers who use the respective channel, not just our five. If we succeed in rejecting H0, the data speak for H1. That doesn't prove H1. Why you proceed this way is explained in the article Understanding hypothesis tests: one idea behind z, t, χ2 and F.

For the calculation, we need a notation. Each individual value in the table is an observation. We write it as ygk. The index g stands for the group, that is, the factor level: g = 1 is phone, g = 2 chat, g = 3 email. The index k counts the responses within the group, from 1 to 5. So the fourth response in the chat is y24 = 6, the fourth by email y34 = 4. G is the number of groups, here 3, K the number of responses per group, here 5.

We also need two kinds of mean. The group mean ȳg (pronounced: y-bar g) is the mean of one group. For phone it is (9 + 8 + 10 + 8 + 10) / 5 = 45 / 5 = 9, for chat (7 + 5 + 8 + 6 + 9) / 5 = 35 / 5 = 7, for email (5 + 6 + 3 + 4 + 7) / 5 = 25 / 5 = 5. The overall mean ȳ without an index is the mean of all 15 responses together: (45 + 35 + 25) / 15 = 105 / 15 = 7.

How do you estimate a value without and with knowledge of the group?

Let's plot the data. The vertical axis shows the satisfaction values. The horizontal axis shows the groups and, within them, the responses side by side. This axis has no other meaning than to assign each value to its group. On the left are the five responses of the phone group, y11 to y15, in the middle those of the chat, on the right those of email, y31 to y35.

Scatter plot with 15 points. Vertical axis satisfaction from 0 to 10, horizontal axis three sections side by side: phone, chat, email, each with five points from left to right for the responses k = 1 to 5. Phone: 9, 8, 10, 8, 10. Chat: 7, 5, 8, 6, 9. Email: 5, 6, 3, 4, 7. Across all three sections, a red horizontal line at 7, labeled overall mean ȳ = 7. In each section, a short teal horizontal line across the width of the group: at 9 (ȳ₁ = 9), at 7 (ȳ₂ = 7, coincides with the red line, teal slightly thicker or dashed on top of it), at 5 (ȳ₃ = 5). Below the points of the email group, small labels y₃₁ to y₃₅.
Figure 1: The 15 responses by channel, the overall mean 7 (red) and the group means 9, 7 and 5 (teal). Made-up practice numbers.

What do we need the overall mean for? Imagine you are asked to estimate how satisfied the next person to rate the service will be. You know nothing about the channel. A natural estimate is then the mean of the observed responses, that is, 7. This value minimizes the sum of squared deviations in our sample: with any number other than 7, that sum would be larger.

But now we have additional knowledge: the three groups have different means. If you know that the next person called, you can use the phone group's sample mean, 9. If they came by email, you can use 5. Using the group means instead of 7 reduces the total squared error in our sample, although it does not guarantee better predictions for new responses. If there were no difference between the channels, this knowledge would be of no use, and ideally all three group means would lie exactly on the overall mean.

But reality is shaped by chance. Even if all customers in the three channels were equally satisfied on average, the means of our sample would differ somewhat by chance. The question is how large the deviation is. In our example, it is clear: 9 and 5 each lie 2 points away from the overall mean of 7. That is a first sign of a difference between the channels.

Statistically, though, it remains to be clarified whether these differences are significant: are they larger than chance easily produces? Whether the channel is the cause, on the other hand, the test doesn't clarify. Other influences can also be behind a difference, for example how difficult the request was, what mood the person was in that day, who was on duty. Or random influences that can't be measured at all. To answer the question of significance, analysis of variance decomposes the variation.

How do you decompose the variation?

Every point deviates from the overall mean. We decompose this deviation into two pieces.

The first piece runs from the overall mean to the mean of the group. It is called the explained deviation. It is explained because we can predict it with knowledge of the group: if you know which channel a contact came through, the group mean fits our data better than the overall mean, measured by the sum of squared deviations. This piece is the same for all points in a group. For every response by phone it is 9 − 7 = +2, for every one in the chat 7 − 7 = 0, for every one by email 5 − 7 = −2.

The second piece runs from the group mean to the point actually measured. It is called the unexplained deviation. We hold other factors and random influences responsible for it. For the third response by email, y33 = 3, it is 3 − 5 = −2.

The same scatter plot as Figure 1 with the red line at 7 and the teal group lines at 9, 7 and 5. At each of the 15 points, two vertical segments side by side: a red segment from the overall mean 7 to the point (total deviation) and a teal segment from the group mean to the point (unexplained deviation, within the group). Next to each group, a black vertical double arrow between the group mean and the overall mean, labeled: phone +2, chat 0 (no arrow, since there is no distance), email −2 (explained deviation, between the groups). Example highlighted: email point y₃₃ = 3 with a red segment of −4, a black piece of −2 and a teal piece of −2. Legend: red total deviation, black between the groups (explained), teal within the group (unexplained). The segments show directed deviations with signs.
Figure 2: For each point, the red total deviation is the sum of the black explained piece between the groups and the teal unexplained piece within the group, each with its sign.

This way, we have decomposed the total deviation of each point from the overall mean into two parts. The idea behind it once more: if you don't know the channel, you estimate with 7. If you know it, you estimate with the group mean, that is, with 9, 7 or 5, and in our data the sum of squared deviations becomes smaller. The explained piece shows how far the group mean lies from the overall mean. The reduction in total squared error is measured across all observations. Whatever deviation from the group means then remains goes back to random or unaccounted-for influences.

What does this look like as a model?

Written algebraically, each observation is made up of three parts: the overall mean, the effect of the group, that is, the explained deviation, and the unexplained deviation, the random influence.

y_{gk} = \mu + \alpha_g + \varepsilon_{gk}

Here, ygk is the value of the k-th response in group g. μ is the mean of all customers, not just our 15. αg (pronounced: alpha g) is the effect of group g, that is, how far its mean in the population deviates from μ. εgk (pronounced: epsilon g k) is the random deviation of the individual response from the mean of its group. We don't know μ and αg. We estimate them from the data: μ by the overall mean ȳ, αg by ȳg − ȳ. What is then left over for the individual response, ygk − ȳg, is called the residual. It is our estimate of εgk.

Let's pick out one point, the fourth response by email, y34 = 4. The overall mean is ȳ = 7. We estimate the effect of group 3 as ȳ3 − ȳ = 5 − 7 = −2: by email, satisfaction in our data is on average 2 points below the overall mean. The residual is y34 − ȳ3 = 4 − 5 = −1: this person is another 1 point below the mean of their group. Together:

y_{34} = \bar{y} + (\bar{y}_3 - \bar{y}) + (y_{34} - \bar{y}_3) = 7 + (-2) + (-1) = 4
Detail of the scatter plot, only the email group large, the other groups faded in the background. Red horizontal line at 7, labeled ȳ = 7. Teal horizontal line at 5, labeled ȳ₃ = 5. The fourth point y₃₄ = 4 large and highlighted. Next to it, three vertical segments: red from 7 to 4, labeled total −3. Black from 7 to 5, labeled ȳ₃ − ȳ = −2. Teal from 5 to 4, labeled residual y₃₄ − ȳ₃ = −1. Below it, the equation y₃₄ = ȳ + (ȳ₃ − ȳ) + (y₃₄ − ȳ₃) = 7 + (−2) + (−1) = 4, the parts in the colors of the segments (ȳ red, ȳ₃ − ȳ black, residual teal).
Figure 3: The fourth response by email, decomposed: overall mean 7, estimated effect of the group −2, residual −1, together 4.

How do you calculate the sums of squares?

So far, we have looked at individual deviations. Now we combine them for all points. If we simply added up the deviations, the upward and downward ones would cancel each other out, and around the mean the sum would always be zero. That's why you square each deviation and only then add them up. The result is called the sum of squares, because that's exactly what it is: a sum of squares. It is abbreviated SS. An appended letter says which deviation is meant.

Total variation, SST. The T stands for total, so SST is the total sum of squares. For each of the 15 points, we calculate the deviation from the overall mean, square it and add everything up. The overall mean is the same for all three groups, namely 7.

SST = \sum_{g=1}^{G} \sum_{k=1}^{K} (y_{gk} - \bar{y})^2

The two summation signs mean: add over all groups g and, in each group, over all responses k. In the example:

  • Phone: (9 − 7)² + (8 − 7)² + (10 − 7)² + (8 − 7)² + (10 − 7)² = 4 + 1 + 9 + 1 + 9 = 24
  • Chat: (7 − 7)² + (5 − 7)² + (8 − 7)² + (6 − 7)² + (9 − 7)² = 0 + 4 + 1 + 1 + 4 = 10
  • Email: (5 − 7)² + (6 − 7)² + (3 − 7)² + (4 − 7)² + (7 − 7)² = 4 + 1 + 16 + 9 + 0 = 30

Together: SST = 24 + 10 + 30 = 64.

Explained variation, SSB. The B stands for between: SSB is the sum of squares between groups. This is about the black pieces from Figure 2, the deviation of the group mean from the overall mean. The overall mean is the same for all points, the group mean for all points of a group. That's why every point of a group has the same black piece, and we don't have to go through each point individually. We take the squared deviation once per group and multiply it by the number of responses in the group, here K = 5:

SSB = \sum_{g=1}^{G} K \cdot (\bar{y}_g - \bar{y})^2

In the example:

  • Phone: 5 · (9 − 7)² = 5 · 4 = 20
  • Chat: 5 · (7 − 7)² = 5 · 0 = 0
  • Email: 5 · (5 − 7)² = 5 · 4 = 20

Together: SSB = 20 + 0 + 20 = 40, or in short 5 · (2² + 0² + 2²) = 5 · 8 = 40.

Unexplained variation, SSW. The W stands for within: SSW is the sum of squares within groups. These are the teal pieces, the deviation of each point from the mean of its group. Now the group mean stays fixed for each group: 9 for phone, 7 for chat, 5 for email.

SSW = \sum_{g=1}^{G} \sum_{k=1}^{K} (y_{gk} - \bar{y}_g)^2

In the example:

  • Phone: (9 − 9)² + (8 − 9)² + (10 − 9)² + (8 − 9)² + (10 − 9)² = 0 + 1 + 1 + 1 + 1 = 4
  • Chat: (7 − 7)² + (5 − 7)² + (8 − 7)² + (6 − 7)² + (9 − 7)² = 0 + 4 + 1 + 1 + 4 = 10
  • Email: (5 − 5)² + (6 − 5)² + (3 − 5)² + (4 − 5)² + (7 − 5)² = 0 + 1 + 4 + 1 + 4 = 10

Together: SSW = 4 + 10 + 10 = 24.

Now we can check that the numbers are right. The sum of squares between the groups and the sum of squares within the groups together make up exactly the total variation. Figure 2 shows that the deviations add up for each point. For their squares, this only holds in the sum over all points, but there it holds exactly:

SST = SSB + SSW \quad\Rightarrow\quad 64 = 40 + 24

Why divide by the degrees of freedom?

Before we get to the test, one step is still missing. The sums of squares depend on the number of observations. For SST and SSW, this is easy to see: every new response adds a new square, and none of them is negative. So these two sums never get smaller, they grow with the number of responses, even if nothing has changed about the channels. SSB, on the other hand, doesn't have to grow, because with every new response the means are recalculated too and can move closer together. In addition, SSB contains only three groups, whereas SSW contains 15 individual responses. Placed side by side in raw form, the two sums would therefore not be comparable.

That's why we turn each sum of squares into something similar to a variance, that is, an average squared deviation. But we don't divide by the number of observations, we divide by the number of degrees of freedom.

What are degrees of freedom? Suppose three numbers are to add up to 10:

x_1 + x_2 + x_3 = 10

We can choose the first number freely, let's say 15. The second, too, let's say −20. It could have been any other number, even a billion or minus a million. But we can no longer choose the third one freely. The first two add up to 15 + (−20) = −5. For the result to be 10, x3 has to be exactly 15, because −5 + 15 = 10. So of three numbers with a fixed sum, only two are free. The number of values that can vary freely is called the number of degrees of freedom, df for short.

With the same reasoning, we determine the degrees of freedom of the three sums of squares.

Total variation. This is about the red deviations of all individual points from the overall mean. Once the overall mean is fixed, 14 of the 15 points can vary freely, the fifteenth follows. So the degrees of freedom are N − 1, the number of all observations minus 1. N can also be written as the number of groups times the number of responses per group, G · K = 3 · 5 = 15. So df = G · K − 1 = 15 − 1 = 14.

Explained variation. This is about the black pieces. Their special feature: within a group, they are always the same. Once we fix the piece for a group, it applies to all five points of that group. So there are basically only three such pieces, one per group. With groups of equal size, the average of the three group means is exactly the overall mean: (9 + 7 + 5) / 3 = 7. Once the overall mean is fixed, we can therefore choose two group means freely, the third is determined. With groups of unequal size, the same holds with a weighted average in which large groups count more. The degrees of freedom are G − 1 = 3 − 1 = 2.

Unexplained variation. This is about the teal pieces, the deviations of the points from the line of their group mean. This line is given for each group. Of the five points in a group, four can vary freely, the fifth follows, just like with the three numbers that add up to 10. So each group has K − 1 = 4 degrees of freedom. And this holds in each of the three groups. If we had four groups, K − 1 would appear four times. The degrees of freedom are therefore G · (K − 1) = 3 · 4 = 12.

Like the sums of squares, the degrees of freedom also add up to the degrees of freedom of the total variation: 14 = 2 + 12.

So far, every group had the same number of responses. In real surveys, that's rare: far more ratings may come in by phone than by email. Then you take each group's own size ng. In SSB, the squared deviation of each group is multiplied by ng instead of K, and the degrees of freedom within the groups are the number of all responses minus the number of groups:

SSB = \sum_{g=1}^{G} n_g \cdot (\bar{y}_g - \bar{y})^2 \qquad df_{\text{within}} = N - G

With groups of equal size, this is the same as above: N − G = 15 − 3 = 12 = 3 · (5 − 1).

The same scatter plot as Figure 2, three channels with five points each, red line at 7, teal lines at 9, 7 and 5. Three labels with brackets. First, above all 15 points, in red: total variation, 15 red deviations from the overall mean, df = N − 1 = 15 − 1 = 14. Second, at the three black double arrows, in black: between the groups, 3 pieces, one per group, two free and one fixed (the third black piece dashed in gray with a lock or the note fixed), df = G − 1 = 2. Third, in each group, in teal: within, 5 teal pieces each, of which 4 are free and the fifth is fixed (in each group, the fifth piece dashed in gray), df = 3 · (5 − 1) = 12. At the bottom, the sum line: 14 = 2 + 12, in the three colors.
Figure 4: Degrees of freedom of the three sums of squares. Total 15 − 1 = 14, between the groups 3 − 1 = 2, within the groups 3 · (5 − 1) = 12, and 14 = 2 + 12.

If we divide each sum of squares by its degrees of freedom, we get the mean squared deviation, called the mean square, MS for short. This makes the two numbers comparable: MSW estimates the variation that chance brings into the responses. If there is no difference between the channels, MSB estimates the same variation.

MSB = \frac{SSB}{G - 1} = \frac{40}{2} = 20 \qquad MSW = \frac{SSW}{G \cdot (K - 1)} = \frac{24}{12} = 2

MSB is the mean square between the groups, MSW the one within the groups. For the total variation, MST = SST / (N − 1) = 64 / 14 ≈ 4.57. Unlike the sums of squares and the degrees of freedom, the mean squares don't add up: 20 + 2 isn't 4.57. For the test, we only need MSB and MSW.

How does the F-test check whether the difference is significant?

With these numbers, we go into the statistical test. We compare two mean squares with each other: the one between the groups, which belongs to our factor, and the one within the groups. Under the null hypothesis and the model assumptions, this ratio follows an F-distribution. Our test statistic is therefore an observed F-value:

F = \frac{MSB}{MSW} = \frac{20}{2} = 10.0

The numerator contains the mean square between the groups, the denominator the one within the groups. So the mean square between the channels is ten times as large as the one within the channels. If there were no difference between the channels, the ratio should be closer to 1, because then both would estimate the same random variation.

When is F large enough? That's what the critical value tells you. The F-distribution has two kinds of degrees of freedom: those of the numerator and those of the denominator. The third quantity is the significance level, which we set before the analysis. We take 5%, so α = 0.05. There is a separate F-table for each significance level. It is two-dimensional: one direction shows the degrees of freedom of the numerator, that is, of the explained variation, the other those of the denominator, that is, of the unexplained variation.

In our case, these are 2 and 12 degrees of freedom. Between the groups: three groups, 3 − 1 = 2. Within: three groups, in each of which four values can vary freely, 3 · 4 = 12. In the F-table for 5%, the critical value at the intersection of 2 degrees of freedom in the numerator and 12 in the denominator is 3.89.

What does the F-distribution look like? Because it describes the ratio of two squared quantities, it can only take positive values. It starts at zero and runs out to the right in a long tail. Its exact shape depends on both degrees of freedom. With 2 degrees of freedom in the numerator, as in our case, it falls from zero onward. With more degrees of freedom in the numerator, it first rises and then falls. The rejection region therefore lies only on the right: F becomes large when the groups lie far apart.

Density of the F-distribution with 2 and 12 degrees of freedom, horizontal axis F from 0 to 12, the curve starts at its highest point at F = 0 and falls to the right in a long tail. Vertical dashed line at F = 3.89, labeled critical value F_crit = 3.89 (α = 5%). The area to its right filled in red, labeled rejection region 5%. At F = 10.0, a clear point or arrow on the axis, labeled our value F = 10.0. The tiny area to the right of 10.0 highlighted darker, labeled p ≈ 0.003. Title: F(2, 12).
Figure 5: The F-distribution with 2 and 12 degrees of freedom. To the right of the critical value 3.89 lies the rejection region of 5%. Our value F = 10.0 lies far inside it.

Now we compare. The observed value 10.0 is clearly larger than the critical value 3.89. So we reject the null hypothesis that all three population means are equal. The data speak for the alternative hypothesis: mean satisfaction is not the same in all three channels. Which channels differ, the F-test doesn't say, we'll clarify that further below.

Instead of the critical value, software usually gives you the p-value straight away. Here, p ≈ 0.0028, rounded 0.003. This means: if there were no difference between the channels in the population, you would get an F-value of 10.0 or larger in only about 3 out of 1,000 samples. The p-value doesn't tell you how likely H0 or H1 is. What exactly it does tell you is explained in the article P-value and statistical significance.

Two things are not said by this result. First, the difference is not proven. Even without any difference, an F-value lands in the rejection region in 5% of cases. Second, the analysis says that the channels differ, but not why. Unlike in an experiment, we didn't assign the customers to the channels at random, they chose for themselves. Perhaps it's precisely those whose request is difficult who write an email. Then the lower satisfaction would be due to the request and not to the channel. From this comparison alone, you can't conclude that the channel causes satisfaction.

All the numbers so far fit into one table. Every statistics program outputs it in this or a similar form:

ANOVA table for the contact channel example

Source of variationSS
sum of squares
df
degrees of freedom
MS
mean square
F
Source of variationBetween groups (channel, explained)4022010.0
Source of variationWithin groups (unexplained)24122
Source of variationTotal64144.57

How strong is the relationship?

Besides the significance test, analysis of variance has a second task: to measure the strength of the relationship. You always look at this number, whether the test turns out significant or not. The effect size is called eta squared, written η2. It puts the explained variation in relation to the total variation:

\eta^2 = \frac{SSB}{SST} = \frac{40}{64} = 0.625

The idea behind it is very simple. If the factor explained all of the variation, SSB would equal SST and η2 would equal 1. Then there would be no deviation left within the groups, every response would lie exactly on the mean of its group. The less the factor explains, the smaller its share of the total variation becomes, all the way down to zero. With η2 = 0, all group means would lie on the overall mean. So η2 always lies between 0 and 1 and is the share of the explained variation in the total variation. Multiplied by 100, it gives the percentage.

In our example, η2 = 0.625. The contact channel explains 62.5% of the variation in satisfaction, the remaining 37.5% (24 out of 64) remain unexplained within the channels.

Is that a lot? That depends on the yardstick you judge by. As for correlations, there are rules of thumb for η2, and each discipline has its own tradition for them. Two of them side by side:

Rules of thumb for η2 in two traditions

Tradition (source)Classification of η2Our η2 = 0.625
Tradition (source)Behavioral sciences (Cohen 1988, pp. 281 and 285)0.01 small · 0.06 medium · 0.14 large effectlarge
Tradition (source)Social sciences with a view to practice (Ferguson 2009, p. 533, Table 1)0.04 minimum for practical significance · 0.25 moderate · 0.64 strong effectmoderate, just below strong

Own compilation based on the sources named. Cohen gives his levels for a related measure, the values for η2 are converted from them and rounded.

The classifications are traditions of the disciplines, not laws. Cohen explicitly calls his levels a stopgap for when experience in a research area isn't sufficient for better values (Cohen 1988, p. 285). Ferguson understands his values as minimum thresholds that should not be applied rigidly (Ferguson 2009, p. 533).

So our value is large according to Cohen and moderate according to Ferguson, just below the threshold for strong. It is high by both yardsticks. But it comes from made-up practice numbers, chosen so that every step can be calculated in your head. On top of that, with small samples, η2 tends to overestimate the relationship. There, chance alone already produces noticeable differences between the groups, and η2 counts them in. Averaged over many samples, η2 therefore comes out too high. Whether the true share for this one result lies below or above 62.5% can't be said from this. So in a report, give the number and not just a label: "η2 = 0.625" is unambiguous, "strong effect" isn't. If you use a label, name the convention along with it.

Taken together, for our example this means: the difference between the channels is significant, F lies far above the critical value, and the channel explains a large part of the variation.

A horizontal bar, labeled total variation SST = 64 = 100%. It is divided: on the left, a black section over 62.5% of the length, labeled between the groups SSB = 40, explained by the channel, 62.5%. On the right, a teal section over 37.5%, labeled within the groups SSW = 24, unexplained, 37.5%. Below it, the formula η² = SSB / SST = 40 / 64 = 0.625. Above it, a red bracket over the whole bar, labeled 100%.
Figure 6: Of the total variation (64), the contact channel explains 40, that is, 62.5%. η2 = 0.625.

The two have to be considered separately. A small p-value can also come out for a small difference, especially with many responses. Then the result is significant, and yet the factor explains only a very small part of the variation. The F-test checks whether the data are consistent with equal means. η2 says how strong the relationship is in the data. Together, they help assess the evidence for a difference and its size. Its practical importance depends on the context.

What does analysis of variance assume?

The F-test relies on three assumptions. First, the responses are independent of each other: each person rates once and doesn't influence the others. Second, the values in each group are approximately normally distributed. With groups of equal size, the test copes well with small deviations from this. Third, the values vary about equally in all groups. In our example, the variance is 1.0 for phone and 2.5 each for chat and email. With only five responses per group, however, it's hardly possible to check reliably whether the values are normally distributed and vary equally.

Equal variation is usually checked with Levene's test. Statistics programs show it as a separate table or row on the homogeneity of variances, and what matters there is the p-value. If it is below 0.05, that speaks against equal variances. If it is above, equal variation is not proven, all the more so with small groups.

If the variation differs clearly, especially with groups of unequal size, you use Welch's analysis of variance. It doesn't assume equal variation and calculates the degrees of freedom of the denominator differently, which is why they're usually not a whole number. For our numbers, it gives F = 11.03 with 2 and 7.58 degrees of freedom, p = 0.006, and so reaches the same decision. For the pairwise comparisons we'll come to in a moment, you then use the Games-Howell test instead of the Tukey test.

Which channels differ?

A significant F-value says: not all three channels are equal. Which ones differ, it doesn't say. That is clarified by subsequent comparisons, so-called post hoc tests, for example the Tukey test (Tukey HSD). It compares every pair of groups and corrects for the fact that several comparisons are run at once. Usually, you only look at the pairs after a significant F-test. But that isn't necessary: the Tukey test itself keeps the error risk for all pairwise comparisons together within bounds and doesn't require a significant F-test.

In our example, only phone differs significantly from email: 9 versus 5, difference 4, p = 0.002. Phone versus chat and chat versus email are 2 points apart each, p = 0.105. With five responses per channel, that isn't enough to reject H0 for these pairs. But it doesn't follow that phone and chat don't differ. The data just aren't sufficient to show it.

If, on the other hand, you simply ran a t-test for phone versus chat, two-sided, with pooled variance and without correction, you'd get p = 0.044, and the difference would be called "significant." This shows why the correction is needed: each additional test creates another opportunity for a false positive.

What if several factors are at work?

So far, we've had only one factor. Often, though, several are at work at the same time, for example the channel and the type of request. Then two-way analysis of variance asks how much variation each factor explains on its own and whether the effect of one factor depends on the level of the other. This is called an interaction, for example when the chat goes down well with simple requests, but not with difficult ones. The idea stays the same: the variation is decomposed, this time into more parts.

How to report the result

In practice, a program does the calculation, for example SPSS, jamovi or R. It outputs an ANOVA table, and the row for the groups (often "Between groups" or labeled with the name of the factor) shows the sum of squares, df, mean square, F and the p-value, labeled "p" or "Sig." depending on the program. For our numbers, it shows p = 0.003. You can calculate η2 yourself from the same table: the sum of squares between the groups divided by the total sum of squares, here 40 / 64 = 0.625. Many programs also output η2 directly, some as "partial eta squared." With only one factor, it is the same as η2.

How to write it up. The result includes the F-value with both degrees of freedom, the p-value and the effect size, plus the group means with their standard deviations. For our example: "Mean satisfaction differed significantly between the contact channels, F(2, 12) = 10.00, p = .003, η2 = .63. Phone M = 9.00, SD = 1.00; chat M = 7.00, SD = 1.58; email M = 5.00, SD = 1.58." M stands for the mean, SD for the standard deviation, η2 = .63 is 0.625 rounded to two decimal places. APA style, which is common in English-language papers, drops the leading zero for p and η2, because they can't be larger than one, so it writes p = .003. Which style applies is set out in the guidelines of your department. If you ran post hoc tests, also say which pairs differ, here: "According to the Tukey test, only phone differed significantly from email, p = .002."

How to read it. p = .003 means: if there were no difference between the channels in the population, you would get an F-value of 10.0 or larger in only about 3 out of 1,000 samples. p doesn't tell you how likely H0 is, and rejecting H0 proves nothing. A non-significant pair such as phone versus chat doesn't mean "no difference." To draw a conclusion about the cause, you need a suitable study design, most reliably random assignment to the channels. This comparison alone isn't enough for that. More on this in the article Understanding hypothesis tests.

With QUESTIONSTAR, you don't have to calculate this by hand: via the MCP connection, an AI assistant can retrieve the responses to your survey and run the analysis of variance for you.

What to ask when others do the calculations for you

Often, the calculations aren't done by the person who has to decide, but by an agency, a research institute or a colleague in management accounting. If the report says "Satisfaction differs significantly by channel," these five questions help you put it into perspective:

  1. How many responses are in each group? With few responses per channel, even a clear difference is uncertain, and with very many, even a tiny one becomes significant.
  2. How large is the difference? Ask for the means with standard deviations and for η2. Significant doesn't mean important.
  3. Which groups exactly differ? A significant F-test only says that not all are equal. Pairwise comparisons show which pairs differ.
  4. Were other differences between the groups considered? If it's mainly people with difficult requests who get in touch by email, the difference may be due to the request and not to the channel.
  5. Who responded? If only some of the customers contacted responded, the result applies to them first of all.

Frequently asked questions about analysis of variance

What is a one-way analysis of variance, explained simply?

It tests whether the means of several groups, formed by a single categorical characteristic, differ so much that chance alone is a poor explanation. To do this, it compares the variation between the groups with the variation within the groups. The ratio, with both divided by their degrees of freedom, is the F-value.

What does the F-value mean?

F is the ratio of the mean square between the groups to the mean square within the groups, MSB / MSW. In the example, F = 10.0: the mean square between the channels is ten times as large. Large values speak against equal means. Whether that is significant is shown by the comparison with the critical value of the F-distribution for the matching degrees of freedom and the chosen significance level, 3.89 in the example.

What does eta squared (η²) tell you?

η2 is the share of the explained variation in the total variation, that is, SSB / SST. It lies between 0 and 1. In the example, η2 = 0.625: the contact channel explains 62.5% of the variation in satisfaction. Significance and strength are two different questions, which is why η2 always belongs in the result. When a value counts as small or large is set by each discipline according to its own tradition. According to Cohen, for example, a value counts as large from 0.14 (Cohen 1988, pp. 281 and 285).

Does a significant result prove that the channel determines satisfaction?

No. A significant F means: the data speak against the null hypothesis that all group means are equal. That proves no difference, let alone its cause. To draw a conclusion about the cause, you need a suitable study design. Random assignment to the groups is the most reliable.

What is the difference between analysis of variance and the t-test?

The t-test compares the means of two groups. Analysis of variance tests whether the means of a factor with any number of levels differ, and at the same time measures the strength of the relationship with η2. How to calculate the t-test is shown in the article The t-test: comparing means the methodologically clean way. Which test fits your data is covered in the guide Which statistical test should I use?.

Why not simply run several t-tests?

With three groups, you'd need three t-tests, with five groups already ten. At 5%, each test has its own risk of finding a difference that doesn't exist, and the risk of being wrong in this way at least once grows with every test. With three independent tests, it would already be 1 − 0.95³ ≈ 14.3%. Analysis of variance tests all groups with a single test at the chosen level. Which pairs differ is then shown by corrected post hoc tests such as the Tukey test. Usually, you only look after a significant F-test, but the Tukey test doesn't require one.

What do I use if my data aren't metric?

If your data can at least be put in an order, so they are at least ordinal, and the groups are independent, the Kruskal-Wallis test is an option. It doesn't compare means but the ranks of the values, and it doesn't need equal distances between the points of the scale. It isn't suitable for purely nominal data such as colors or brands. For our numbers, with the correction for ties, it gives H(2) = 8.98, and the approximate p-value is 0.011. That is the same decision, only on a slightly different question: it compares ranks, not means. More in the guide Which statistical test should I use?.

Sources

Backhaus, K., Erichson, B., Plinke, W. and Weiber, R. (2018). Multivariate Analysemethoden. Eine anwendungsorientierte Einführung. 15th ed. Berlin: Springer Gabler. DOI 10.1007/978-3-662-56655-8. Chapter 4, Varianzanalyse (analysis of variance), pp. 163-201. The lecture on which this article is based follows the structure of this textbook: model, decomposition of variation, significance test and eta squared. The worked example in this article is made up specifically for it.

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates. Chapter 8, p. 281 (η² from the effect size f, equation 8.2.19) and p. 285 (small, medium and large f-values of 0.10, 0.25 and 0.40 as a convention for when experience in the research area is lacking).

Ferguson, C. J. (2009). An Effect Size Primer: A Guide for Clinicians and Researchers. Professional Psychology: Research and Practice, 40(5), 532-538. DOI 10.1037/a0015808. p. 533, Table 1 (squared association indices r², R², η²: 0.04 minimum for practical significance, 0.25 moderate, 0.64 strong effect).

NIST/SEMATECH (n.d.). e-Handbook of Statistical Methods, section 7.4.3 "Are the means equal?" with the subsections 7.4.3.1 "One-way ANOVA overview" to 7.4.3.4 "1-Way ANOVA calculations". National Institute of Standards and Technology, itl.nist.gov, accessed November 15, 2026. Covers the model, the decomposition of the sums of squares, the ANOVA table and the F-test of one-way analysis of variance.

Related articles:

Dr. Paul Marx

Author: Dr. Paul Marx
Dr. Paul Marx is the founder and Managing Director of QUESTIONSTAR and was a professor of marketing at the University of Siegen.