Delphi method: steps, an example and how to define consensus

How a Delphi survey works: expert panel, rounds, feedback and consensus criteria. With a two-round example, pros and cons, and tips for your thesis.

In the Delphi method, also called the Delphi technique, a selected panel of experts assesses a topic over several rounds. After each round, everyone sees a summary of the group's answers and can reconsider their own judgment, without knowing who on the panel said what. That way you can pool expert judgment on questions that don't have a settled answer yet. Agreement at the end shows that this panel converged under these rules. It isn't proof that the panel is right.

Let's start with an example right away. A university wants to prioritize the requirements for a new research data service. A panel of experts is asked which requirements matter and under which conditions they're feasible. After a first survey, the participants see a summary and rate again. In the end, the answers are more alike. Has the process found the right plan?

The convergence is, first of all, a result of this panel and this procedure. To interpret it, you need to know who took part, what feedback they got and what counted as agreement. That's why these decisions are made before the first round, not when you read the results. Below I'll walk you through the process round by round, show you a sample questionnaire for two rounds, and work through a calculation that shows how easily an apparent consensus comes about. The university case and all numbers are made-up teaching examples.

What is the Delphi method?

Let's begin with an everyday example. Five friends are estimating how long the drive to their vacation spot will take. Each of them writes a number on a slip of paper without talking to the others. Then someone collects the slips and reads them out: the estimates range from five to nine hours, most of them at six. One person has added a note: "On Fridays there's always a traffic jam at the highway interchange outside town." Now everyone gets to estimate again. Whoever knows the route well may stick with their number. Whoever hadn't thought of the traffic jam adjusts. Nobody has to justify themselves in front of the others, and the loudest person at the table doesn't automatically win.

That's exactly the core of the Delphi method. Möhring and Schlütz, in a German textbook on surveys, define the Delphi survey as a "repeated survey of experts with controlled feedback" (my translation), used mainly for forecasts (Die Befragung in der Medien- und Kommunikationswissenschaft, 2019, section 5.4, p. 168, publisher DOI). The aim is to reach a group consensus "or a reasoned dissent" on a topic.

By the way, the name refers to the Oracle of Delphi in ancient Greece (Nasa, Jain and Juneja 2021, introduction). The method was developed in the context of technological and military developments and was originally used for technology assessment (Möhring and Schlütz 2019, p. 168).

The four defining features

Aichholzer, drawing on Rowe and Wright (1999), names four basic features (The Delphi Method, in Bogner, Littig and Menz, Interviewing Experts, 2009, chapter 12, p. 252, publisher DOI):

1. Anonymity: the participants don't see who answered what.
2. Iteration: at least two rounds.
3. Controlled feedback: the research team summarizes the answers and feeds them back.
4. Statistical aggregation: usually the median or the mean, with information on the spread.

Anonymity and feedback are meant to allow a kind of virtual discussion that isn't distorted by status, group pressure or rhetoric (Aichholzer 2009, pp. 252-253). So the procedure is more than the same questionnaire twice. The feedback is a planned intervention in how people form their judgments. A change can mean that someone has taken an argument into account they hadn't known before. It can also come from a wish to move closer to the group. The change alone doesn't tell you which.

The word "anonymous" also needs precision. Are individual answers hidden only from the other panel members? Does the research team know who is who? What could a detailed comment give away? Your description of the procedure should explain these levels instead of promising anonymity across the board.

Types of Delphi

Aichholzer (2009, p. 259), following Rauch (1979), distinguishes three basic types. The classical Delphi aims at a group opinion, for example in a forecast. The policy Delphi makes the views of stakeholders visible. The decision Delphi prepares decisions. In a real-time Delphi, there are no separate rounds. Participants can view updated, aggregated group feedback online and revise their judgments without waiting for separate rounds (Aichholzer 2009, p. 270). This article is about the Delphi with separate rounds, the kind you'll often see in theses and projects.

How to conduct a Delphi survey step by step

Möhring and Schlütz (2019, pp. 169-170) describe survey waves with analysis in between. In practice, that comes down to seven steps.

1. Define the aim and the subject. A forecast, a prioritized list, a shared wording or a map of open disagreements? These aims don't need the same rules.
2. Choose the expert panel. People are selected deliberately for how close they are to the question, not at random. Plan a reserve for dropouts. Expert panel
3. Set the consensus rule and the end point. What counts as agreement, and after how many rounds do you stop? Consensus criteria
4. Run round 1. Everyone fills out the questionnaire on their own. It usually contains rating scales, often open questions too, and a question on how confident someone is in their judgment (Möhring and Schlütz 2019, p. 169).
5. Analyze and prepare the feedback. The answers are tabulated and presented so that everyone gets an impression of the group's opinion (Möhring and Schlütz 2019, p. 169). Designing the feedback
6. Run round 2 and further rounds if needed. Everyone rates again. People who deviate clearly often explain why, and these reasons go into the next round (Möhring and Schlütz 2019, pp. 169-170). Sample questionnaire
7. Report the result. The report states the consensus rule, the respondents per round, the statements that were changed and the dissent that remains. Documenting

How many rounds?

According to Aichholzer (2009, p. 252), it's usually two rounds, rarely more. According to Möhring and Schlütz (2019, p. 170), a third round usually follows, and whether a fourth is worth it depends on the aim.

Larger changes in judgment, according to Rowe and Wright (1999), are limited to the first rounds, and motivation drops noticeably with each additional round (Aichholzer 2009, p. 267). My advice: set a stopping rule before you start. Here's an example. Two rounds are fixed. A third round is held only for requirements that are contested after round 2, meaning at least 50% but less than 75% of the valid ratings are in the top two categories. After that you stop, and whatever is still contested you document as dissent. Reaching a threshold doesn't prove that the judgments are stable, by the way. With ten ratings, a single changed answer can move the share by 10 percentage points, as example A2 below shows.

Choosing the expert panel

In the university example, requirements are to be ranked for a later decision. So the team doesn't survey a representative sample of all researchers. It deliberately selects people with relevant knowledge and different perspectives: researchers from several disciplines, IT support, the library, data protection and a doctoral student as a user. Choose experts whose experience is relevant to the intended use. Representativeness isn't the main concern (Möhring and Schlütz 2019, p. 170).

Who counts as an expert depends on the question (Möhring and Schlütz 2019, p. 169, footnote 84). The Austrian Technology Delphi deliberately included people from companies, public administration, associations and user representatives. The reason: specialization narrows the view, and experts tend toward excessive optimism (Tichy 2004, cited in Aichholzer 2009, p. 260). Justify the composition of your panel. "Experts from our network" isn't a justification. What experience was required, which role is missing, what conflicts of interest are there?

How many experts does a Delphi study need?

There's no fixed number. According to Parentè and Anderson-Parentè (1987), there's no fixed upper limit, and the recommended minimum is ten people, counted after dropouts (cited in Aichholzer 2009, p. 262). In medical research, published studies range from 10 to 1,000 participants according to a review by Nasa, Jain and Juneja (2021, section "Size"), with most between 10 and 100. For the small working groups in the Austrian Technology Delphi, about two dozen people were invited because up to 50% dropout was expected. Between 14 and 23 took part per topic area (Aichholzer 2009, p. 262).

Planning for dropout between rounds

Panels lose participants. Many Delphi surveys have substantial attrition between rounds 1 and 2. Drilling (2000) reads this to mean that some experts only want to give their own statement and then drop out (cited in Möhring and Schlütz 2019, p. 170).

Let's calculate with assumed values. The university wants ten ratings in the final round. It expects 80% of those invited to take part in round 1 and 85% of them to take part in round 2 as well. Arithmetically, that's 10 / (0.8 × 0.85) ≈ 14.7, rounded up to 15 invitations. On average, you can then expect 10.2 participants in round 2. If everyone decides independently whether to take part, though, the probability of really getting at least ten is only ≈ 66%. If you need at least ten, you need an extra reserve. Use participation rates from comparable projects, and err on the side of caution.

What helps against dropout? Give good reasons for taking part and state the time each wave takes realistically, because after the first wave everyone knows how much work it is (Möhring and Schlütz 2019, p. 170). In the Austrian Technology Delphi, this included a personal cover letter, a reminder after three weeks and follow-up phone calls (Aichholzer 2009, p. 266).

Rate importance and feasibility separately

"This requirement makes sense and can be implemented quickly" contains two judgments. A person can find it important and still think it's hard to implement. That's why the team asks separately and, for feasibility, names a time frame and the resources available.

There's also an option to skip a requirement for lack of knowledge. Lack of knowledge isn't coded as low importance. Every summary states how many people actually rated the item. In the Austrian Technology Delphi, too, only those who reported at least average expertise on a statement were included in the analysis (Aichholzer 2009, pp. 261 and 267).

Check your terms as well. If everyone finds "documentation" important, one person may be thinking of a guide to the first login and another of a technical manual.

As a rule of thumb, not as a limit: a Delphi statement should be concrete and no longer than 20 words, and an expert group shouldn't rate more than 25 statements (Parentè and Anderson-Parentè 1987, cited in Aichholzer 2009, p. 262). For help wording individual statements, see how to write good survey questions.

Consensus criteria: define agreement before round 1

Your own planning note can contain five decisions:

A review by Diamond and colleagues shows why this planning note matters. They examined 100 randomly selected English-language Delphi studies from 2000 to 2009. Only 72 gave a definition of consensus. The most common definition was percent agreement (25 studies), with 75% as the median of the thresholds used. In 86 studies, the authors concluded that consensus had been reached, but only 42 of them had set a threshold in advance. In only 23 studies did the end of the process depend on whether consensus had been reached (Diamond et al., 2014, abstract).

A commonly used threshold like 75% is therefore not a universal mark of quality. In their review of medical research, Nasa, Jain and Juneja (2021, section "Consensus") report that the percentage thresholds used range from 50% to 97% and are often chosen arbitrarily. You need to justify the rule for your own task.

Common measures for consensus

For the consensus rule, three building blocks are available, which you can use on their own or combined.

• Share in named categories: for example, the share of valid ratings in the top two categories, "somewhat important" and "very important".
• Central tendency: median or mean. The median is the value in the middle when you sort all ratings by size.
• Spread: for example, the interquartile range (IQR). It tells you how wide the middle half of the ratings is. The smaller it is, the closer the judgments are to each other.

In the made-up project, the team reports the share of clearly named categories and the full distribution. The threshold is justified on substantive grounds and isn't shifted afterward so that a favored requirement just makes it. For contested requirements, the protocol states whether they get another round, are reworded or are classified as unresolved. Then you can see from the final list how it came about.

Feedback between rounds: keep the minority visible

For round 2, everyone receives the same group summary: the rating distribution, the number of valid responses and short, anonymized reasons for different positions. If the protocol provides for it, each participant also receives their own previous rating. That's my suggestion, not the only form. In the description by Möhring and Schlütz (2019, p. 169), a person's own answer isn't returned at all, only the group average.

A single measure can hide an important split. Take the made-up requirement "training on anonymizing interview data". Twelve people rate it on a scale from 1 ("very unimportant") to 5 ("very important"). Three give it a 5, three a 4, three a 2 and three a 1. Nobody picks the middle.

The mean is (3 × 5 + 3 × 4 + 3 × 2 + 3 × 1) / 12 = 36 / 12 = 3.0. The median is also 3.0, namely halfway between the sixth rating (2) and the seventh (4). Both measures say "neither important nor unimportant", although nobody gave exactly that answer. In fact, there are two camps. Feedback that only reports "median 3" invites everyone to meet in a middle that doesn't exist.

So show the full distribution. Summarize the reasons according to transparent rules, and don't pick only the comments that support your own preference. It may then turn out that one camp is thinking of disciplines with many interviews and the other of disciplines that work only with measurement data. Then the dissent itself is the result.

Word the invitation to rate again neutrally, as in the example below, and ask for reasons in both cases, whether someone changes their rating or keeps it. A person who deviates shouldn't have to justify themselves more than a person who joins the majority. The next round doesn't test whether someone has learned the right group answer.

Delphi survey example: questionnaire for round 1 and round 2

Here's what the questionnaire for the university could look like. It's my own made-up teaching example, not a validated scale. For each requirement, the questionnaire asks two questions: "How important is this requirement for the new service?" and "How easy is it to implement within the next twelve months with the current staff?" Both scales have five points and are symmetric with a neutral midpoint. "Can't judge" is set apart.

Importance: very unimportant · somewhat unimportant · neither important nor unimportant · somewhat important · very important

Feasibility: very difficult to implement · somewhat difficult to implement · neither easy nor difficult to implement · somewhat easy to implement · very easy to implement

(set apart) can't judge

Round 1

Each requirement has an open field: "What influenced your rating the most?" At the end, the questionnaire asks: "Which requirement is missing from this list?" This open question brings in aspects the research team hadn't thought of. A1 ("documentation") is deliberately worded too vaguely, because that happens so often in practice. The section below shows how A1 changes in round 2.

Round 2: what the feedback looks like

In round 2, each rating is preceded by a short block with the result of the first round. For A2, it would look like this:

A2: Advice on the data management plan is available within five working days.

Result of round 1, importance (12 valid ratings):
very unimportant 1 · somewhat unimportant 2 · neither important nor unimportant 2 · somewhat important 4 · very important 3
Median: somewhat important

Reasons from round 1, shortened:
"Without a reliable turnaround time for advice, researchers risk missing grant application deadlines."
"We cannot provide this advice within five working days with the current staffing. A realistic deadline would be better."

Please review your assessment in light of this feedback. You can change it or keep it. In either case, please give a brief reason for your rating.

The two reasons stand for two positions, not for the majority. Look at the second one: it's really about feasibility, not importance. You'll find more symmetric scales in our list of common Likert scales.

By the way, in QUESTIONSTAR you set up the experts as a contact list and invite each person by email with a personal link. Reminders go only to those who haven't answered yet. You create the second round as a copy of the first questionnaire, send it to the same contact list and put a text block with the feedback in front of each requirement. Use a stable participant key for both rounds, and before sending, check with a test contact that it comes out identically in both exports. You'll find more on using QUESTIONSTAR at universities under surveys in academic research.

You can also send an anonymous survey with personal links and reminders. In the results, you then don't see who answered what. So decide before round 1 which variant you need. With the anonymous setting, you get the group feedback for each round, but linking the rounds person by person isn't provided for. If you want to return each person's earlier rating to them, or compare only the same people as in the example below, you need the variant in which the answers are linked through the participant key. Anonymity then applies toward the other participants, not toward the research team. Say so openly in the invitation.

Worked example: what really happened after round 2

Before round 1, the team decided: a requirement counts as "important by consensus" if at least 75% of the valid ratings are in the top two categories. The 75% is a choice for this example, not a general rule. "Can't judge" doesn't count as a valid rating.

As calculated above, 15 people were invited. 12 took part in round 1, that's 12 / 15 = 80%. 10 of those 12 took part in round 2, so 10 / 12 ≈ 83%. The team was slightly below its assumption and just made the ten ratings it needed.

Requirement A2: consensus reached?

The middle column can only be calculated if the answers of the same people are linked across both rounds.

At first glance, this is a success: the share rises from ≈ 58% to 80%, the 75% threshold is passed, and the spread is cut in half.

Let's take a closer look. Two people didn't answer round 2. They happened to be the two with the lowest ratings, a 1 and a 2. If you compare only the ten people who answered both rounds, their share was already 70% in round 1. So the increase of ≈ 22 percentage points splits into two parts:

• ≈ 12 percentage points (from ≈ 58% to 70%) result solely from the two lowest-rating participants dropping out.
• 10 percentage points (from 70% to 80%) come from changed judgments. In the linked individual data of our teaching example, we can see that exactly one person changed their rating from 3 to 4. Everyone else kept their rating.

The interquartile range is also cut in half by this one change alone. With ten ratings, it reacts strongly to individual people.

In short: if agreement rises between two Delphi rounds, that can be due to changed judgments or to dissenting people dropping out. You can only tell the two apart with linked answers from the same people. In example A2, ≈ 12 of the ≈ 22 percentage points are due to dropout and 10 to a single changed rating. Formally, A2 meets the rule. The report should still say: "In round 2, the two people with the lowest ratings from round 1 no longer took part."

And feasibility? In round 2, the same ten people also rated A2 on that dimension: twice "very difficult to implement", four times "somewhat difficult to implement", twice "neither easy nor difficult to implement" and twice "somewhat easy to implement". So only 2 / 10 = 20% think advice within five working days is somewhat easy to implement, 6 / 10 = 60% think it's difficult, and the median is "somewhat difficult to implement". The result for A2 is therefore: important by the agreed rule, but difficult to implement with the current staff. Before a decision, the resources need to be checked.

Requirement A5: which denominator?

For A5, the central data catalog, all 12 people answered in round 1. Four of them chose "can't judge". Of the 8 valid ratings, 6 are in the top two categories. That's 6 / 8 = 75%, exactly at the threshold.

If the team had accidentally used all 12 answers as the denominator, the share would be 6 / 12 = 50%. If it had counted "can't judge" as the midpoint, the share would also be 50%. Same answers, two different conclusions. That's why you define the denominator in your protocol, and the report states for each requirement how many people actually rated it.

How to calculate this with your data

Set up "can't judge" as a separate opt-out answer, not as a point on the scale. In QUESTIONSTAR, an answer like that isn't counted in the mean. Check with a test response that it isn't in the denominator of your agreement share either, otherwise you get the error from example A5. Calculate the IQR and compare matched responses across rounds outside QUESTIONSTAR. There are two ways to do this.

Give each expert the same personal ID in both rounds. It appears in the answers of both rounds and serves as the key. The ID of an individual response, on the other hand, is new in each round and isn't suitable for linking.

One option is to connect QUESTIONSTAR via MCP to an AI assistant such as ChatGPT or Claude. Ask the assistant to merge both rounds using this personal ID and calculate the statistics. A prompt could read, for example: "Merge round 1 and round 2 using the respondents' personal ID. For question A2, calculate for round 1 and round 2 the share of 'somewhat important' and 'very important' answers, the median and the interquartile range. Don't count 'can't judge', and give the number of valid ratings for each round, once for everyone and once only for the people who answered both rounds." The calculation is then done by the assistant, not by QUESTIONSTAR. The second way: export the raw data of both rounds as a CSV, Excel or SPSS file, merge the files using the personal ID and calculate in Excel, SPSS or R. Check one result by hand. There are several ways to compute quartiles. The table above follows Excel (QUARTILE.INC). State in your report how you calculated.

Questions marked as personal data are not transferred to the assistant when the protection setting is on. Unmarked answers, especially free text, can still contain personal information. Before transferring, check which answers go to the AI provider and whether its contract terms suit your data.

Advantages and disadvantages of the Delphi method

What are the advantages of a Delphi survey? Compared with a group discussion, Möhring and Schlütz (2019, p. 170) list as advantages that there are no dominant personalities, that unimportant information is filtered out and that the respondents don't have to be in the same place at the same time. I'd add: because each person answers on their own, quiet voices get heard too.

How good are the results? Rowe and Wright (1999) reviewed 27 evaluation studies. Overall, Delphi groups in the review tended to be more accurate than statistically aggregated individual judgments and unstructured discussion groups, but not consistently better than other structured methods. That agreement increases over the rounds was found practically everywhere. Why exactly remained open (Aichholzer 2009, pp. 253-254).

There are disadvantages, too. Möhring and Schlütz (2019, pp. 170-171) name the pressure toward consensus created by the feedback, the rigid survey format and the large amount of time required. On top of that come objections that are still being debated today (Möhring and Schlütz 2019, p. 171, Aichholzer 2009, p. 253):

• The consensus can be an artifact of the procedure, because it's convenient to go along with the majority.
• The research team aggregates the answers and therefore has a lot of influence. The feedback can also be manipulated.
• The deliberate selection of experts can steer the result, and specialists can be biased in favor of their own field.
• Response often drops noticeably in the last rounds.

My advice: Delphi is worth it when there are no reliable data yet and relevant knowledge is distributed across several experts. If you want to know how widespread an opinion is in a population, you need a different kind of survey.

A Delphi study for your bachelor's or master's thesis

Many students consider a Delphi study because the procedure is structured and doesn't require travel. Because of the workload involved, though, Möhring and Schlütz (2019, p. 172) consider repeated survey designs "only suitable to a limited extent" (my translation) for theses.

Clarify these points with your supervisor beforehand:

• Research question: is it about experts assessing uncertain matters?
• Panel: who counts as an expert, how will you reach these people, and how many dropouts do you expect?
• Rounds and schedule: two fixed rounds are a realistic frame. For each round, plan time in the field, reminders and the analysis in between.
• Consensus rule: which measures, which threshold, which denominator, and why?
• Anonymity: what exactly do you promise the participants, and how will you link the rounds if you need to?

Keep the list of statements short, because every expert is doing you a favor, again, in every round. How to structure the methods section as a whole is covered in How to write your thesis.

Documenting changes and unresolved points

If "provide documentation" becomes "provide a short guide to the first login, in English and the local language" in round 2, the subject has been clarified. The new rating no longer measures the same statement. Save both versions together with the reason for the change.

The number and composition of respondents per round also belong in the report. Example A2 has shown why.

If there's no agreement at the planned end, document the positions, the conditions and the open questions that remain to be checked. A sentence like "contested within the panel under these conditions" can be more useful for the decision than a forced consensus. Delphi organizes expert judgments. What you conclude from them is your own responsibility as the research team.

Frequently asked questions about the Delphi method

How many experts does a Delphi study need?

There's no universal minimum. Parentè and Anderson-Parentè recommend at least ten people after possible dropouts, cited in Aichholzer (2009, p. 262). What matters are the research question, the perspectives represented and availability. The panel section above shows how to calculate how many experts to invite.

How many rounds does a Delphi survey have?

Usually two or three (Aichholzer 2009, p. 252, Möhring and Schlütz 2019, p. 170). Set the number and the rule for an additional round before you start. More above.

Is a Delphi survey anonymous?

Individual responses are normally anonymous to the other panel members. That's one of the basic features (Aichholzer 2009, p. 252). Whether the research team also can't see who answered what is up to you. With the anonymous setting in QUESTIONSTAR described here, linking the rounds person by person isn't provided for (more on this). Describe exactly in the invitation who can see what.

When is consensus reached?

When the rule you set before round 1 is met. There's no generally valid threshold, and 28 of 100 studies examined gave no definition at all (Diamond et al., 2014). Alongside the measure, always report the distribution and the number of respondents per round.

Is the Delphi method suitable for a bachelor's thesis?

Yes, if you plan the effort realistically. Möhring and Schlütz (2019, p. 172) consider repeated survey designs only suitable to a limited extent for theses. With a small panel that's easy to reach, two rounds and a short list of statements, a Delphi study is feasible, though. More on this above.

How is it different from expert interviews and focus groups?

In an expert interview, you talk to each person individually and usually in an open format. In a focus group, several people discuss at the same time. A Delphi study collects expert judgments over repeated rounds with controlled feedback. It may start with open-ended questions and later move to structured ratings. The example in this article starts with predefined statements and also allows participants to suggest additional requirements. It has the most in common with group discussion, without the participants having to meet (Möhring and Schlütz 2019, p. 170).

Sources

Aichholzer, G. (2009). The Delphi Method: Eliciting Experts' Knowledge in Technology Foresight. In A. Bogner, B. Littig and W. Menz (eds.), Interviewing Experts (chapter 12, from p. 252). Basingstoke: Palgrave Macmillan. DOI 10.1057/9780230244276. Used: pp. 252-271.

Möhring, W. and Schlütz, D. (2019). Die Befragung in der Medien- und Kommunikationswissenschaft. Eine praxisorientierte Einführung [Surveys in media and communication studies: a practical introduction] (3rd ed.). Wiesbaden: Springer VS. In German. DOI 10.1007/978-3-658-25865-8. Sections 5.4 and 5.5, pp. 168-172.

Diamond, I. R. et al. (2014). Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. Journal of Clinical Epidemiology, 67(4), 401-409. DOI 10.1016/j.jclinepi.2013.12.002, PubMed.

Nasa, P., Jain, R. and Juneja, D. (2021). Delphi methodology in healthcare research: How to decide its appropriateness. World Journal of Methodology, 11(4), 116-129. DOI 10.5662/wjm.v11.i4.116.

Cited in Aichholzer (2009): Rowe and Wright (1999), Parentè and Anderson-Parentè (1987), Rauch (1979), Tichy (2004). Cited in Möhring and Schlütz (2019): Drilling (2000).

I wish you an engaged panel and every success with your Delphi study!

Related articles:

• How to write your thesis
• How to write good survey questions
• Common Likert scales: examples with labels

Dr. Paul Marx