What a p-value tells you and what it doesn't, why statistically significant doesn't mean important, and how precise your survey results are. With calculator.
In short: the p-value tells you how likely a result like yours, or a more extreme one, would be if the null hypothesis were true (that is, if there were in fact no difference or no association in the population) and the other assumptions of the test held. It doesn't tell you how likely it is that your hypothesis is true, and it doesn't tell you how large or important an effect is (Wasserstein and Lazar 2016, pp. 131-132). For most survey results, a confidence interval is therefore more useful: it shows the range in which the true value plausibly lies, for example "62% satisfied, ±4.8 percentage points," and it requires a random sample (Jacob et al. 2019, p. 71).
Confidence interval for a proportion
How precise is a percentage from your survey? Normal approximation, everything is computed in your browser.
Enter n and the proportion or the count.
Applies to random samples. For small populations (sample of 10% of the population or more), the interval is somewhat too wide, see Calculating sample size.
Your entries stay in your browser. No data is transmitted.
How to use the calculator. Enter how many valid responses you have for a question, and either the proportion in percent or the number of people who chose a particular answer. Choose the confidence level. 95% is the usual choice. The calculator shows the margin of error in percentage points and the interval in which the true proportion plausibly lies. If fewer than ten people chose the answer, or fewer than ten did not choose it, it warns you, because the calculation becomes unreliable. The calculator works for proportions, not for means. Your entries stay in your browser, and no data is transmitted. How the calculator works is explained below under Confidence interval for a proportion.
Statistical significance, the p-value and the confidence interval belong together, but they're often confused. In this article I'll answer five questions in turn:
- What is a p-value, and what does "statistically significant" mean?
- What are the most common misreadings of the p-value?
- Why isn't a significant result automatically important?
- How precise is a percentage from my survey?
- When does all of this apply, and what changes with many tests or an online survey with an open link?
You may know the situation: satisfaction has risen from 58% to 62%, and in the meeting someone immediately asks, "Is that significant?" If you say no, people often hear "So nothing has changed." If you say yes, they hear "Great, the program works." Both go too far. This article shows what's behind the question, what answer you can give, and why a clearly reported interval usually gives you more to work with than an asterisk. How to calculate a t-test or a chi-square test step by step is covered in the articles The t-test and The chi-square test. This article is about the logic behind them and about reading the results correctly.
What is a p-value?
Every significance test follows the same pattern. You set up a null hypothesis, usually "no difference" or "no association," and an alternative hypothesis, its opposite. You decide in advance what risk of error you're willing to accept. That's the significance level α. Then you calculate a test statistic from your data and decide whether to reject the null hypothesis (Schumann 2019, pp. 206-207; Malhotra and Birks 2007, p. 511).
The p-value is the probability, under the null hypothesis and the other assumptions of the model, of getting a result that deviates from the null hypothesis at least as much as yours does (Wasserstein and Lazar 2016, p. 131). The smaller it is, the worse your data fit the null hypothesis, provided the assumptions behind the calculation hold (ibid.).
A result is called statistically significant when the p-value is below the significance level chosen in advance. Common levels are 0.05, 0.01 or 0.001, that is 5%, 1% or 0.1% (Schumann 2019, p. 206). Tables often mark this with asterisks: one asterisk for 0.05, two for 0.01 (Weinreich and von Lindern 2008, p. 103). Which level is appropriate depends on what an error would cost (Malhotra and Birks 2007, p. 513; Schumann 2019, p. 208, n. 21).
Worked example: has satisfaction increased?
Here's a made-up example that we'll carry through the whole article. An online store surveys 400 randomly selected customers every year. The question is:
"Overall, how satisfied are you with our service?"
very dissatisfied · somewhat dissatisfied · neither satisfied nor dissatisfied · somewhat satisfied · very satisfied
Last year, 232 of 400 answered "somewhat satisfied" or "very satisfied." That's 58%. This year it's 248 of 400, or 62%. The difference is 4 percentage points. Is that larger than the difference we would expect from random sampling alone?
The null hypothesis is: the share of satisfied customers hasn't changed in the customer base. We test it with the z-test for two proportions from independent samples, here different people in the two years, as described by Malhotra and Birks (2007, pp. 530-531):
"Two-sided" (also called two-tailed) means that we count deviations in both directions, because it wasn't settled in advance whether satisfaction would rise or fall. A one-sided test checks only one direction, and it's appropriate only if that direction was your hypothesis before the survey (Malhotra and Birks 2007, p. 512).
A p-value of 0.25 means: if nothing had changed in the customer base, surveys like these with 400 people each would show a difference of 4 percentage points or more, in one direction or the other, about a quarter of the time. So a difference of this size is nothing unusual with 400 responses per year. At a significance level of 5%, the result is not significant. With this number of responses and proportions around 60%, only a difference of about 7 percentage points would be significant.
What does that mean for the meeting? Not "nothing has changed." Instead: "Satisfaction is 4 percentage points higher than last year. But with 400 responses each year, a difference like this wouldn't be unusual even if nothing had actually changed." What exactly you may conclude from a non-significant result is the topic of the next section.
Two errors every test can make
A test can be wrong in two ways. It can reject the null hypothesis although it's true. That's a Type I error. If the null hypothesis is true, this happens with probability α, the significance level. Or it can fail to reject the null hypothesis although it's false. That's a Type II error, and its probability is called β (Malhotra and Birks 2007, p. 513; Schumann 2019, p. 208). The probability of detecting a difference that really exists is called the power of the test, 1 − β (Malhotra and Birks 2007, p. 513).
| In truth no difference (H0 is true) | In truth a difference (H0 is false) | |
|---|---|---|
| Test rejects H0 ("significant") | Type I error, probability α | correct, probability 1 − β (power) |
| Test doesn't reject H0 | correct, probability 1 − α | Type II error, probability β |
My own illustration based on Malhotra and Birks 2007, p. 513, and Schumann 2019, p. 208.
In my course "Principles of Survey Research," I explain the two errors with a criminal trial (slide 188 and slide 189): the null hypothesis is "the defendant is innocent," the Type I error is convicting an innocent person, and the Type II error is letting a criminal go free.
The two errors are linked. If you choose a very small α to be on the safe side, you're more likely to miss real differences (Schumann 2019, p. 208). And if you have few responses, you have little power. For the online store with 400 responses per year, a real increase from 58% to 62% would come out significant in only about one in five such surveys (power ≈ 0.21, my own calculation). So the 4 percentage points may be real even though the test isn't significant.
What doesn't the p-value tell you? Six common misreadings
In 2016, the American Statistical Association (ASA) published a statement on the p-value, because it's misread so often. The statement sums up its position in six principles (Wasserstein and Lazar 2016, pp. 131-132):
- "P-values can indicate how incompatible the data are with a specified statistical model."
- "P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone."
- "Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold."
- "Proper inference requires full reporting and transparency."
- "A p-value, or statistical significance, does not measure the size of an effect or the importance of a result."
- "By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis."
The following table applies them to sentences that actually appear in theses and reports.
| Sentence in the report | Is that right? | What's correct |
|---|---|---|
| Sentence in the report"p = 0.03, so there's a 3% probability that the null hypothesis is true." | No | The p-value is calculated on the assumption that the null hypothesis is true. It says nothing about how probable that hypothesis is (principle 2, p. 131). |
| Sentence in the report"p = 0.03, so there's a 3% probability that the result is due to chance." | No | The p-value doesn't measure that either (principle 2, p. 131). It tells you how well the data fit the "chance alone" model, not how probable that model is. |
| Sentence in the report"p < 0.05, so the difference is important." | No | Significance measures neither the size of an effect nor its importance (principle 5, p. 132). |
| Sentence in the report"p > 0.05, so there is no difference." | No | A large p-value is not evidence in favor of the null hypothesis (principle 6, p. 132). The null hypothesis can be rejected, but it can never be accepted on the basis of a single test (Malhotra and Birks 2007, p. 512). |
| Sentence in the report"p = 0.049 is a finding, p = 0.051 is not." | No | A conclusion doesn't become "true" on one side of the threshold and "false" on the other (principle 3, p. 131). A p-value near 0.05 taken by itself is only weak evidence against the null hypothesis (principle 6, p. 132). |
| Sentence in the report"We ran 30 comparisons, two were significant, and those are the ones we report." | Problematic | If you select only the significant results, the p-values become practically uninterpretable. Report how many analyses you ran (principle 4, pp. 131-132). More on this under Many tests. |
Principles from Wasserstein and Lazar 2016, examples and explanations my own.
The fourth row deserves a second look, because it does the most damage in practice, especially with small groups. Weinreich and von Lindern (2008, pp. 101-102) describe a customer survey in which the 56-to-65 age group stood out dramatically in the chart. None of it was significant, because only 13 of the 610 respondents belonged to that age group. The right conclusion in that case is "We don't know," not "The older customers are no different."
Instead of reporting only "significant" or "not significant," the ASA points to approaches that emphasize estimation over testing, such as confidence intervals (Wasserstein and Lazar 2016, p. 132).
Significant doesn't mean important: effect size and practical significance
Whether a result comes out significant depends not only on how large the difference is, but also on how many responses you have. Let's take the online store and change only the number of responses:
| Scenario | Share last year | Share this year | Difference | Responses per year | p (two-sided) | Significant (5%)? |
|---|---|---|---|---|---|---|
| ScenarioA | 58% | 62% | 4 percentage points | 400 | ≈ 0.25 | no |
| ScenarioB | 58% | 62% | 4 percentage points | 4,000 | ≈ 0.0003 | yes |
| ScenarioC | 61% | 62% | 1 percentage point | 20,000 | ≈ 0.04 | yes |
Made-up example, my own calculation (z-test for two proportions).
A and B show the same difference, but only B is significant. C is significant although the difference is just one percentage point. This is exactly what the ASA means: "Any effect, no matter how tiny, can produce a small p-value if the sample size or measurement precision is high enough, and large effects may produce unimpressive p-values if the sample size is small or measurements are imprecise" (Wasserstein and Lazar 2016, p. 132). Malhotra and Birks (2007, p. 531) show the reverse picture: 73% versus 40% is a substantial difference, but with 15 respondents in each group it isn't significant.
So whether a difference matters in practice isn't decided by the test. It's decided by your question. One more percentage point of satisfaction may be worth a lot of money to a mail-order company with millions of customers, and mean nothing for a department of 40 employees. That's why you should always report the size of the difference in units people understand (percentage points, in the case of proportions) and, next to the p-value, a measure of strength:
- For proportions: the difference in percentage points with its confidence interval (see Comparing two groups or two years).
- For means: Cohen's d as well, explained in the article The t-test.
- For crosstabs: phi or Cramér's V. The chi-square test only checks whether there is an association, not how strong it is (Schumann 2019, p. 209; Malhotra and Birks 2007, p. 521). If you double every count in a crosstab, chi-square doubles, but the association stays the same (slide 202). How to calculate these measures is shown in the article The chi-square test.
Rule to remember: The p-value answers "How unusual would this result be if there were in fact no difference?" The effect size answers "How large is it?" Whether it's important is something you answer yourself, with your decision in mind.
How precise is my result? The confidence interval for a proportion
Most survey results are proportions: 62% are satisfied, 35% would recommend the product, 18% use the service daily. The most important question about them isn't "significant or not?" but: how far can this value be from the true proportion in the population? The confidence interval gives the answer.
A confidence interval is a range around your result, calculated from the sample and tied to a confidence level, usually 95%. The confidence level describes the method: it's the share of intervals calculated this way that would contain the true value across a very large number of random samples (Schumann 2019, p. 191). Half the width of the interval is called the margin of error.
The formula
For a proportion p from n valid responses, the interval is (Schumann 2019, p. 193; Malhotra and Birks 2007, pp. 440-441):
It's the same formula you use in the article Calculating sample size to determine the number of responses you need, just read in the other direction. There, you set the margin of error and solve for n. Here, you have n and calculate the margin of error.
Worked example: 248 of 400
For the 62% satisfied customers in the online store example:
At 90%, the interval is narrower (58.0% to 66.0%), and at 99% it's wider (55.7% to 68.3%). A higher confidence level means a wider interval (Schumann 2019, p. 192). A quick rule of thumb for 95% and proportions around 50%: margin of error ≈ 1 / √n, so about ±5 percentage points with 400 responses (slide 152).
What does the 95% mean? It describes the method, not your individual interval. If you repeated the survey many times with new random samples, roughly 95 out of 100 intervals calculated this way would contain the true proportion. "Roughly," because the formula is an approximation. You don't know whether your interval is one of them. The true value either lies within it or it doesn't (Schumann 2019, p. 191, n. 37). So the sentence "There's a 95% probability that the true value lies between 57.2% and 66.8%" is, strictly speaking, not correct, even though you'll find this kind of wording in textbooks too (for example, Malhotra and Birks 2007, p. 437). In a report, you're on safe ground with this wording:
"62% of respondents are somewhat or very satisfied (95% confidence interval 57.2% to 66.8%, n = 400, random sample from the customer database)."
How wide is the interval for a given number of responses?
Four horizontal bars around the same dot at 62 percent. With 100 responses, the interval runs from 52.5 to 71.5 percent, with 400 from 57.2 to 66.8, with 1,000 from 59.0 to 65.0, and with 1,600 from 59.6 to 64.4 percent. The more responses, the narrower the bar.
95% confidence intervals for a proportion of 62%, simple random sample, large population. My own calculation.
To cut the width of the interval in half, you need four times as many responses (Schumann 2019, p. 192, n. 43): 400 instead of 100, 1,600 instead of 400. The same applies to subgroups. If you report results by department, calculate the interval with the number of responses from that department. If the sales team has 25 responses and 15 of them are satisfied, the result is 60% ±19.2 percentage points, so somewhere between 41% and 79%. That gives you only a rough idea of the proportion. And because only ten people did not choose "somewhat satisfied" or "very satisfied," the case only just meets the rule of ten, so the approximation remains rough here (see the next section). The rule of thumb of at least 20 to 25 cases per subgroup (Jacob et al. 2019, p. 72) is a lower limit, not a target.
When the formula stops working
The formula is an approximation based on the normal distribution. It becomes imprecise when the proportion is close to 0 or 100% and there are few responses. One rule of thumb says that n × p × (1 − p) should be greater than 9 (Bosch 2012, p. 178, cited in Schumann 2019, p. 193). Better known is the rule that at least 10 people should have given the answer and at least 10 should not have given it, that is n × p ≥ 10 and n × (1 − p) ≥ 10 (Diez et al. n.d., section 6.1). I'll call it the rule of ten here. The two rules of thumb are not equally strict: for proportions in the middle, the rule of ten is more lenient than Bosch's rule, while near 0 or 100% it can be stricter. At a proportion of 50%, Bosch's rule requires more than 36 responses, while the rule of ten requires only 20. The calculator above warns according to the rule of ten. If your case is only just above it, like the sales team with 15 of 25 (which no longer passes Bosch's rule), read the interval as a rough guide.
Here's an example of a borderline case: 3 of 30 respondents reported a problem. That's 10%. The formula gives ±10.7 percentage points, so the calculated interval runs from −0.7% to 20.7%. A negative proportion is impossible, and with only three mentions the approximation is unreliable. Statistics software offers more accurate methods for cases like this. For the report, the honest statement is often enough: "3 of 30 respondents."
Two more limits. The formula assumes a simple random sample from a large population. If you survey a large part of a small group, say 80 of 100 employees, the interval is too wide, in this example more than twice as wide as necessary. The correction for finite populations is covered in the article Calculating sample size. And the formula captures only the random error of sampling, not ambiguous questions or systematic nonresponse (Malhotra and Birks 2007, p. 85).
And it applies to proportions. For a mean, such as average satisfaction on a scale from 1 to 5, the interval is: mean ± z × sample standard deviation / √n. Strictly speaking, you should use the appropriate critical value from the t-distribution instead of z. With about 100 responses or more, the difference is usually very small (Schumann 2019, pp. 191-192, n. 40). When you compare two means, the t-test tests the difference.
Comparing two groups or two years
Back to the year-over-year comparison. The two intervals look like this:
| Year | Share satisfied | 95% confidence interval |
|---|---|---|
| YearLast year | 58% (232 of 400) | 53.2% to 62.8% |
| YearThis year | 62% (248 of 400) | 57.2% to 66.8% |
| YearDifference | +4 percentage points | −2.8 to +10.8 percentage points |
Made-up example, my own calculation.
For the comparison, what counts is the interval of the difference in the last row. It ranges from a slight decrease to a clear increase and includes zero. That fits the non-significant test from above, but it tells you more: both an increase of 10 percentage points and a decrease of 2 percentage points lie within this interval. So the data don't yet allow a precise statement about the direction and size of the change. That turns "not significant" into a statement you can work with.
Be careful when you look at two separate intervals: the fact that they overlap is not a test. Two intervals can overlap even though the difference is significant. An example: 58% (53.2% to 62.8%) and 65% (60.3% to 69.7%), with 400 responses each. The intervals overlap between 60.3% and 62.8%, yet the test of the difference gives p ≈ 0.04. So calculate the difference directly, with the z-test for two proportions (Malhotra and Birks 2007, pp. 530-531), with the chi-square test for crosstabs, or with the t-test for means.
The calculation here applies to two independent samples, that is, different people in the two years. If you survey the same people both times, you need a test for paired samples.
How are sample size, margin of error and significance related?
All three depend on the same thing: the n under the square root. As the sample grows, confidence intervals become narrower (Jacob et al. 2019, p. 71), and at the same α the risk of missing a difference that exists goes down (Malhotra and Birks 2007, p. 513). Two practical rules follow from this.
Before the survey, decide how precise your result needs to be and calculate the number of responses from that. The article Calculating sample size and the sample size calculator show you how. If you want to compare changes or groups, the margin of error for a single value isn't enough. To detect a real increase from 58% to 62% as significant in four out of five surveys, the online store would need around 2,350 responses per year instead of 400 (my own approximate calculation, power 80%, α = 0.05, two-sided). Demonstrating small differences is expensive.
After the survey, calculate with the actual number of responses for each question. With missing answers, it can be smaller than the number of completed questionnaires.
A large sample is no substitute for good selection, though. The best-known example is the Literary Digest poll of 1936: 2.4 million responses and still the wrong election winner, because the selection was flawed (Jacob et al. 2019, pp. 71-72, in detail in the article Sampling methods). No confidence interval in the world would have revealed that error.
Many tests, many false positives
Surveys tempt you into many comparisons. Twenty questions, each comparing women with men: that's 20 tests. If the null hypothesis is true for all of them and the tests are independent, then at α = 0.05 one of them will still come out significant on average, and the probability of at least one such false positive is 64% (Bland and Altman 1995, p. 170). It rises quickly with the number of tests:
| Number of tests | Probability of at least one false positive (α = 0.05) |
|---|---|
| Number of tests1 | 5% |
| Number of tests5 | 23% |
| Number of tests10 | 40% |
| Number of tests20 | 64% |
| Number of tests40 | 87% |
My own calculation: 1 − 0.95^k, assuming that all null hypotheses are true and the tests are independent.
What helps:
- Set your hypotheses in advance. Before the analysis, write down which two or three comparisons really interest you. Those are the ones you test. Everything else is exploration and is reported as such.
- Report everything you calculated, not just the significant results (see the last row of the table above).
- Be stricter with many tests. The simplest correction, the Bonferroni method, divides the significance level by the number of tests (Bland and Altman 1995, p. 170). With 20 tests, the threshold is then p < 0.0025 instead of p < 0.05. That's strict and more likely to miss real differences, but it protects against spurious findings. If the tests depend on each other, which is almost always the case with questions put to the same people, the correction becomes even stricter than necessary (ibid.).
- Confirm surprises. An unexpected significant finding from a large number of comparisons is an exploratory result. Report it as a lead, and test the hypothesis that follows from it in the next survey.
Does this also apply to my online survey with an open link?
P-values and confidence intervals rest on the assumption that the respondents were selected at random. Only then can results be generalized to the population with statistical methods, and only then can confidence intervals be meaningfully calculated (Jacob et al. 2019, p. 71). The American Association for Public Opinion Research (AAPOR) also states that the margin of sampling error applies only to probability-based surveys and not to opt-in online surveys (AAPOR n.d., p. 1). If you drew a random sample from a customer database and sent personal invitations to the people drawn, the precondition is met in principle, as long as nonresponse doesn't distort the picture.
If you shared the link on social media or put it on your website, the participants selected themselves. You can still run the calculations, but the result doesn't automatically apply to a population. In that case, report tendencies, describe how the participants were recruited, and give p-values or intervals at most under the explicit assumption that the sample behaves like a random sample. I explain which sampling methods exist and what each one allows in the article Sampling methods. What "representative" means is covered in Representative sample.
Decision path. If the respondents were not selected at random, report tendencies. Otherwise, check whether at least ten people gave the answer and at least ten did not. If not, report counts. If so, report the proportion with its confidence interval. For comparisons, calculate the difference with its 95 percent confidence interval and p-value. If the comparison wasn't planned in advance, report it as an exploratory lead and state the number of all tests. If it was planned, check whether the confidence interval of the difference includes zero. A two-sided test at a significance level of 5 percent goes with a matching 95 percent confidence interval. Close to the significance threshold, different approximation methods can disagree. With a correction for multiple tests, the test and the interval must use the same correction. If the interval includes zero, report the difference with its interval and state that there is insufficient evidence to conclude that there is a difference. If it excludes zero, ask whether the difference is large enough for a decision. If it's too small, report it as small. Otherwise, report it as a finding with its size, interval and number of responses.
A p-value alone is not a statement. It belongs together with the number of responses, the size of the difference and the type of sample.
My own illustration based on Wasserstein and Lazar 2016, pp. 131-132, and Jacob et al. 2019, p. 71.
P-values and tests with QUESTIONSTAR
In QUESTIONSTAR, you run the common tests right in the tool. It supports t-tests for paired and independent samples, correlations (Pearson's r, Spearman's rho, Kendall's tau) and regression analyses, with statistical significance reported (Features). You create crosstabs in the reports. The chi-square value, the p-value and Cramér's V are shown there as well.
If you work with an AI assistant connected to QUESTIONSTAR via the MCP interface, it can start paired t-tests (two questions, same respondents), correlations and linear regressions on your survey data. The calculation runs server-side in QUESTIONSTAR. The assistant calls it and explains the result. For the paired t-test, you get the mean difference with its 95% confidence interval in addition to the p-value. The assistant can calculate other comparisons itself from the responses, for example comparisons between two groups. In that case, ask it to name the method and the numbers of responses, and have it keep count of the tests it runs. Questions you've marked as personal data are not passed to the assistant when the corresponding setting is switched on. Answers without that mark, especially free-text answers, can still contain personal data. So check beforehand which questions the assistant gets to see.
For the confidence interval of a proportion, take the number of valid responses and the proportion from the report on your question and enter both in the calculator above. For your own analyses, export the raw data as an SPSS, CSV or XLSX file.
Checklist: reporting significance and precision correctly
- Sampling method stated (random sample from which list, or self-selection via an open link)
- Number of responses given for each result, and for subgroups the number in that group
- With a random sample and sufficient counts: proportions reported with a confidence interval or margin of error and the confidence level. With self-selection: the limited generalizability stated explicitly instead
- Differences given in percentage points or as an effect size, not just as "significant"
- P-values given exactly (p = 0.03 instead of just p < 0.05, very small values as p < 0.001 instead of p = 0.000), significance level set in advance
- "Not significant" worded as "insufficient evidence to conclude that there is a difference," not as "no difference"
- Number of all tests run stated, and with many tests either corrected or labeled as exploration
- With fewer than 10 responses in one of the two groups (answer chosen or not chosen): counts reported instead of an interval
Frequently asked questions about the p-value and statistical significance
What is a p-value in one sentence?
The p-value is the probability of getting a result like the one observed, or a more extreme one, if there is in fact no difference or association and the assumptions of the test hold. A small p-value means the data fit "no difference" poorly (Wasserstein and Lazar 2016, p. 131).
What does p < 0.05 mean?
p < 0.05 means: if the null hypothesis were true and the other assumptions of the test held, a result at least this extreme would occur in fewer than 5% of comparable samples. If you set the significance level α = 0.05 in advance, the result is statistically significant at that level. The significance level is the threshold you choose before the analysis, and the p-value is what your data produce. It doesn't mean there's a 95% probability that your hypothesis is true, and it doesn't mean the effect is large or important (Wasserstein and Lazar 2016, pp. 131-132).
What does "not significant" mean?
The p-value is not below the significance level set in advance, so the null hypothesis is not rejected. That isn't evidence that there is no difference (Wasserstein and Lazar 2016, p. 132; Malhotra and Birks 2007, p. 512). Especially in small groups, even real differences often fail to reach significance.
What is the difference between statistical significance and practical significance?
A result is statistically significant when its p-value is below the significance level chosen in advance. The p-value is calculated assuming that the null hypothesis is true and the other assumptions of the test hold. Practical significance says whether it matters for your decision. With very large samples, even tiny differences become significant, and with small samples, large differences can go unnoticed (Wasserstein and Lazar 2016, p. 132). So always report the size of the difference as well.
How do I calculate a confidence interval for a proportion?
Standard error = √(p × (1 − p) / n), margin of error = z × standard error, with z = 1.96 for a 95% confidence level, and confidence interval = proportion ± margin of error. For 248 of 400 (62%), that gives ±4.8 percentage points, so 57.2% to 66.8% (Schumann 2019, p. 193). As a rule of thumb, at least ten people should have given the answer and at least ten should not have given it. Close to that limit, the approximation remains rough.
Can I run significance tests on an online survey with an open link?
You can run them, but the precondition of random selection isn't met. Drawing conclusions about the population with statistical methods requires random selection (Jacob et al. 2019, p. 71). So report results as tendencies and state the assumption explicitly.
What is the difference between a confidence interval and a confidence level?
The confidence interval is the range around your result, for example 57.2% to 66.8%. The confidence level, for example 95%, describes the method: across a very large number of repetitions, about 95 out of 100 intervals calculated this way would contain the true value (Schumann 2019, p. 191).
Sources
AAPOR, American Association for Public Opinion Research (n.d.). Margin of Sampling Error/Credibility Interval. Statement, aapor.org (PDF), accessed November 11, 2026. p. 1 (margin of sampling error only for probability-based surveys).
Bland, J. M. and Altman, D. G. (1995). Multiple significance tests: the Bonferroni method. BMJ, 310(6973), 170. DOI 10.1136/bmj.310.6973.170. p. 170 (20 tests, Bonferroni correction, dependent tests).
Diez, D. M., Barr, C. D. and Çetinkaya-Rundel, M. (n.d.). OpenIntro Statistics. Online edition at LibreTexts (stats.libretexts.org), no edition stated, accessed November 11, 2026. Section 6.1 "Inference for a Single Proportion" (condition of at least 10 successes and 10 failures).
Jacob, R., Heinz, A. and Décieux, J. P. (2019). Umfrage. Einführung in die Methoden der Umfrageforschung [Surveys: an introduction to the methods of survey research] (4th ed.). Berlin/Boston: De Gruyter Oldenbourg. In German. pp. 71-72 (random selection, confidence intervals, Literary Digest, number of cases per subgroup).
Malhotra, N. K. and Birks, D. F. (2007). Marketing Research. An Applied Approach (3rd European edition). Harlow: Pearson Education. p. 85 (total error), pp. 437-441 (confidence intervals, proportions), pp. 511-513 (hypothesis testing, one-tailed and two-tailed tests, types of error, power), p. 521 (significance and strength of association), pp. 530-531 (test for two proportions).
Marx, P. (2026). Principles of Survey Research. Online course, chapter 4, slide 152, and chapter 5, slides 188, 189 and 202. QUESTIONSTAR. Slide 152, slide 188, slide 189, slide 202.
Schumann, S. (2019). Repräsentative Umfrage. Praxisorientierte Einführung in empirische Methoden und statistische Analyseverfahren [Representative surveys: a practical introduction to empirical methods and statistical analysis] (7th ed.). Berlin/Boston: De Gruyter Oldenbourg. In German. pp. 191-193 (confidence intervals for means and proportions), pp. 206-209 (significance test, alpha and beta error, strength of association).
Wasserstein, R. L. and Lazar, N. A. (2016). The ASA's Statement on p-Values: Context, Process, and Purpose. The American Statistician, 70(2), 129-133. DOI 10.1080/00031305.2016.1154108. pp. 131-132 (definition and six principles).
Weinreich, U. and von Lindern, E. (2008). Praxisbuch Kundenbefragungen [Customer surveys: a practical handbook]. München: mi-Fachverlag, FinanzBuch Verlag. In German. pp. 101-103 (significance tests in practice).
I hope your next survey gives you results you can describe not only in terms of significance, but also in terms of precision and practical importance.
Related articles:
- Calculating sample size: technical details explained simply
- The t-test: comparing means the methodologically clean way
- Sampling methods: random, quota and cluster sampling
Dr. Paul Marx
Author: Dr. Paul Marx
Dr. Paul Marx is the founder and Managing Director of QUESTIONSTAR and was a professor of marketing at the University of Siegen.
