---
url: "https://questionstar.com/blog/comparing-repeated-surveys/"
title: "Repeated surveys: are the results comparable?"
description: "The score went up. But was the same thing measured? Five checks before the trend chart: wording, target population, field conditions, panel or cross-section, metric."
lastmod: "2026-11-19"
type: "blog"
---

# Did something change, or did we ask differently?

*The second survey shows a higher score. Did things improve, or was it measured differently?*

The score went up. But was the same thing measured? Five checks before the trend chart: wording, target population, field conditions, panel or cross-section, metric.

A university surveys students in autumn and again in spring. A business measures service satisfaction regularly. After the second survey, the chart shows a higher value.

The first question is often why things improved. Before explaining the difference, we need to ask a methodological question: can the two values meaningfully be compared?

A time trend requires more than two numbers placed next to each other. Interpretation depends on what was measured at each point and who answered. This piece is about what should stay the same between two waves and how you can still improve a questionnaire.

## A small edit can create a different measurement

Consider a fictional example. In the first wave, a team asks about satisfaction “with the service overall”. In the next wave it asks about satisfaction with “your most recent service interaction”. The response scale is unchanged, but the reference has shifted.

A single interaction and an overall experience need not receive the same assessment. The difference may therefore relate to the new reference. Without further investigation, the team should not present it as improvement in an unchanged metric.

Response categories, order and context can also affect measurement. Asking about the last problem immediately before an overall assessment creates a different frame from beginning with positive experiences (more on this in the article on [order effects](/blog/order-effects-in-surveys/)). For comparisons over time, the rule is therefore: keep wording, framing and procedure as similar as possible.

## Who was reached at each point?

An unchanged questionnaire does not guarantee comparable respondent groups. A survey during examination season may reach different people and capture different experiences than one at the beginning of term. A business may see the balance between new and long-standing customers shift.

Document the target population, access to the survey, invitation, field period and relevant respondent characteristics.

The checks should fit the study. They do not promise that every possible bias can be repaired afterwards. Similar demographic profiles do not mean that groups match on every characteristic relevant to the research question.

## The same people or new cross-sections?

A [panel](/blog/panel-survey/) aims to survey the same people repeatedly. That can make individual changes examinable, but creates its own requirements. If people drop out and differ systematically from those remaining, the basis for comparison changes.

Repeated cross-sectional surveys draw new samples at different points. They can examine changes at group level when selection and measurement are suitable. That does not establish that the same individuals changed their minds.

These designs answer different questions. Choose and justify the design before the first wave. Linking personal responses also raises [privacy](/blog/data-protection-in-online-surveys/) and confidentiality requirements. It should not be introduced casually.

## Make questionnaire improvements visible

A confusing or flawed questionnaire should not stay unchanged forever simply to preserve a series. Improvements need a documented version.

Record changes to items, options, order and analysis rules. Depending on the research question, a planned transition study can help compare the old and new versions. Whether it is useful depends on the design and resources.

The report should show where interpretation is limited. “We introduced a revised item in spring” is more informative than an apparently seamless line across a measurement change.

## Five checks before the trend chart

Before a team speaks of an improvement or a decline, a quick comparison is worth it:

1. Are the subject, wording and response options comparable?
2. Was a comparable target population reached through an appropriate selection process?
3. Were field period, invitation route and context sufficiently similar? If not, are the differences documented and taken into account in the interpretation?
4. Are panel attrition or differences in sample composition addressed appropriately?
5. Are the metric, percentage base and treatment of missing answers defined consistently?

An observed difference may indicate real change. It may also reflect uncertainty or changes in the research process. A before-and-after comparison alone does not prove that a particular intervention caused the difference.

Author: Dr. Paul Marx
Dr. Paul Marx is the founder and Managing Director of QUESTIONSTAR and was a professor of marketing at the University of Siegen.

## Related articles

- [panel-survey](/blog/panel-survey)
- [customer-satisfaction-score-csat](/blog/customer-satisfaction-score-csat)
- [customer-surveys-overview](/blog/customer-surveys-overview)
- [presenting-survey-results](/blog/presenting-survey-results)

*Author: Dr. Paul Marx — see [About](/about).*
