If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Corresponding author. Prof. Dr. Marlen Niederberger. Department of Research Methods in Health Promotion and Prevention, University of Education Schwaebisch Gmuend, Oberbettringer Straße 200, 73525 Schwäbisch Gmünd, Germany.
Delphi techniques are conducted across different subfields in the health sciences. The reporting practices of studies using Delphi techniques vary, and current reporting guidelines for Delphi techniques focus on individual subfields of the health sciences or on different aspects of research and are therefore of limited applicability. The aim of this article was to identify similarities, differences, and possible shortcomings of existing Delphi reporting guidelines and to draft an initial proposal for a comprehensively applicable reporting guideline.
Methods
A systematic literature search for reporting guidelines on Delphi studies was performed in existing data resources based on databases in the health sciences (Scopus, MEDLINE, CINAHL, Epistemonikos) including publications from 2016 to 2021. In June 2021, we conducted an additional search in PubMed and included further studies by contacting experts of the scientific Delphi expert network (DeWiss). Title and abstract screening of articles was performed, followed by a full-text screening of the articles included. We qualitatively and quantitatively evaluated, compared and contrasted the reporting guidelines identified using content analysis and discussed the results among the members of the Delphi expert network.
Results
We retrieved ten health science articles with reporting guidelines for Delphi studies. In analyzing them, we identified nine main categories (Justification, Expert panel, Questionnaire, Survey design, Process regulation, Analyses, Results, Discussion, Methods reflection & Ethics). The current reporting guidelines vary significantly, with only the aspect of consensus appearing in all of them. Frequency distributions show that most of the subcategories are only addressed in individual articles (e. g., meeting of participants, proceeding with the survey method, transfer of the results, validation, prevention of bias) and that epistemological foundations of the Delphi technique are rarely mentioned or reflected on. We drafted an initial proposal for Delphi reporting guidelines for the health science sector.
Discussion
A well-justified position concerning epistemological foundations of Delphi studies is necessary to make the quality of the process assessable and, along with the reporting of the process, to classify and compare study results. This will increase the acceptance of both the method in the health science sector and the results in medical practice. A Delphi reporting guideline must, above all, take into account the diversity of variants, subfield-related objectives and application areas, and their modifications of the Delphi technique in order to be comprehensively applicable in the health sciences.
Conclusion
The results of our methodological review do not provide a final reporting guideline. The newly developed proposal is intended to encourage discussion and agreement in further analyses.
Zusammenfassung
Hintergrund
Delphi-Verfahren werden in den Gesundheitswissenschaften disziplinübergreifend durchgeführt. Bisher ist die Berichterstattung von Studien mit Delphi-Verfahren sehr unterschiedlich, und die vorhandenen Reporting Guidelines zu Delphi-Verfahren weisen unterschiedliche gesundheitswissenschaftliche teilbereichs- oder forschungsspezifische Schwerpunktsetzungen auf und sind daher nur eingeschränkt anwendbar. Ziel dieses Beitrags ist es, Gemeinsamkeiten, Unterschiede sowie etwaige Mängel bestehender Reporting Guidelines für Delphi-Studien herauszuarbeiten und einen ersten Vorschlag für eine umfassend anwendbare Delphi-Reporting-Guideline zu formulieren.
Methode
Wir führten eine systematische Literaturrecherche nach Reporting Guidelines zu Delphi-Studien in zwei Datenbeständen mit Publikationen zu Delphi-Studien aus den Jahren 2016 bis April 2021 aus Datenbanken im gesundheitswissenschaftlichen Bereich (Scopus, MEDLINE, CINAHL, Epistemonikos) durch. Ergänzend erfolgte eine Suche in PubMed im Juni 2021 und weitere Studien wurden über die Kontaktierung des wissenschaftlichen Delphi Expert*innennetzwerks (DeWiss) eingeschlossen. Das Screening der Artikel erfolgte anhand von Titel und Abstract, gefolgt von einem Volltext-Screening der eingeschlossenen Artikel. Die identifizierten Reporting Guidelines wurden mithilfe einer Inhaltsanalyse qualitativ und quantitativ ausgewertet, gegenübergestellt und verglichen und im Autor*innenteam und mit weiteren Expert*innen des Delphi-Netzwerks diskutiert.
Ergebnisse
Es wurden zehn Artikel mit Reporting Guidelines für Delphi-Studien identifiziert. Durch die Inhaltsanalyse konnten neun Hauptkategorien (Begründung, Expertenpanel, Fragebogen, Erhebungsdesign, Prozessgestaltung, Analysen, Ergebnisse, Diskussion, Methodenreflexion & Ethik) gebildet werden. Die vorliegenden Berichterstattungsleitlinien unterscheiden sich erheblich und nur der Aspekt der Konsensfindung wird in allen genannt. Die quantitative Analyse zeigt, dass einzelne Subkategorien nur in einzelnen Artikeln adressiert (bspw. Treffen der Teilnehmer*innen, Umfrageverlauf, Ergebnistransfer, Validierung, Prävention von Verzerrungen) und epistemologische Grundlagen der Delphi-Methode kaum angesprochen und reflektiert werden. Ein erster Vorschlag für eine Delphi-Reporting-Guideline im Gesundheitsbereich wurde formuliert.
Diskussion
Eine Positionierung im Hinblick auf epistemologische Fragen ist notwendig, um die Prozessqualität bewertbar zu machen und mithilfe der Berichterstattung über den Delphi-Prozess Studienergebnisse vergleichen und einordnen zu können. Dies trägt zur Akzeptanz der Methode in den Gesundheitswissenschaften und von deren Ergebnissen in der Praxis im Gesundheitswesen bei. Darüber hinaus muss eine Delphi-Berichterstattungsleitlinie die Vielfalt der Varianten, teilbereichsspezifischen Ziele und Anwendungsbereiche sowie die jeweiligen Modifikationen der Delphi-Technik berücksichtigen, um eine umfassende Anwendung in den Gesundheitswissenschaften zu ermöglichen.
Schlussfolgerung
Die Ergebnisse unseres methodischen Reviews stellen keine endgültige Richtlinie für die Berichterstattung dar. Der neu entwickelte Vorschlag sollte in weiteren Analysen reflektiert und konsentiert werden.
Delphi techniques are group discussion procedures in which complex issues about which uncertain and incomplete knowledge exists are assessed by experts in an iterative and structured process [
]. They are conducted in different subfields of health science (e.g., clinical patient care, public health and health promotion, health service research, medical education), particularly to identify consensus [
]. Evidence-based knowledge, practical knowledge, and sometimes everyday knowledge based on life-world experiences are brought together in a structured process. The goal is to develop consensual guidelines or standards that are then used to support decision-making and action in the health sector. The methodological literature describes five characteristics of Delphi techniques [
Experts with specific knowledge are interviewed, usually while preserving their anonymity.
2.
The survey is conducted in at least two Delphi rounds, more than three Delphi rounds being the exception.
3.
The survey instrument is usually a standardized questionnaire, often with open-ended questions to capture arguments and reasoning horizons.
4.
The statistical analysis is usually based on descriptive calculations.
5.
From the second Delphi round onwards, the experts receive feedback on the results of the previous round in addition to the questionnaire and can reconsider or revise their judgments.
There are other Delphi variants in addition to the traditional Delphi approach. The most important variants are the following:
•
A real-time Delphi in which expert judgments are reflected back online and in real time; i.e., respondents can make their judgments several times, and there are no separate individual Delphi rounds [
Through a Delphi process, diverging expert assessments and opinions become transparent and ultimately–at least this is the claim–resolved and consented upon. This applies in particular to clinical guidelines [
Methodischer Standard für die Entwicklung von Qualitätsindikatoren im Rahmen von S3-Leitlinien – Ergebnisse einer strukturierten Konsensfindung (Methodological Standard for the Development of Quality Indicators within Clinical Practice Guidelines – Results of a structured consensus process).
White Paper for global palliative care advocacy: recommendations from a PAL-LIFE expert advisory group of the Pontifical Academy for Life, Vatican City.
A national consensus management pathway for paediatric inflammatory multisystem syndrome temporally associated with COVID-19 (PIMS-TS): results of a national Delphi process.
]. The findings of Delphi studies often have far-reaching consequences for health and patient care. In particular, the epistemic claim of a purely knowledge- or experience-based discourse determines the probability of a relatively large acceptance of the results in healthcare practice. A transparent and critical presentation of the Delphi process and the generation of results can contribute considerably to quality assurance and increasing the acceptance of Delphi techniques.
Reporting quality of Delphi techniques
The implementation and data quality of Delphi techniques are rarely discussed or evaluated epistemologically, methodologically or empirically [
] describe the common publication practice of Delphi studies in the health sciences in an overview of twelve systematic reviews (with a total of 883 publications). This overview shows that most publications report on central aspects of Delphi techniques, namely the selection of experts, the definition of consensus, the number of rounds and the response rates. However, it also reveals a certain heterogeneity in reporting quality. For example, a clear statement concerning the applied Delphi variant is found in the systematic reviews in 61% to 88% of the Delphi articles examined. Also varied was the availability of information about participants’ expertise or the selection strategy. In a systematic review from the palliative field, almost one in three articles showed deficiencies in the presentation of questionnaire development [
Using the Delphi technique to determine which outcomes to measure in clinical trials: Recommendations for the future based on a systematic review of existing studies.
] of the articles in the systematic reviews. Possible reasons for these differences in reporting quality have not yet been systematically investigated. They are probably due to field-specific practices, publisher requirements, and the lack of a comprehensively and generally accepted reporting guidelines for studies using Delphi techniques.
Reporting guidelines for Delphi procedures in the health sciences
Reporting guidelines are becoming more and more established in the health sciences in order to standardize and improve the reporting of empirical studies. They typically include a list of criteria that should be addressed in a report based on a particular research method. In particular, the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) network has set out to improve the quality of scientific publications in the health sciences by providing a collection of reporting guidelines. Reporting guidelines now exist for various types of studies and their application has become established internationally (e.g., CONSORT, CONsolidated Standards of Reporting Trials [
]) varies between (and also within) subfields of the health sciences and determines the design of the Delphi process, e.g. in terms of group size and level of heterogeneity or the type of knowledge (level of expertise) and how it is asked for [
]. There are various proposals for reporting guidelines for Delphi studies in the health science sector; one has already been published in the EQUATOR network [
]. However, it remains unclear if these guidelines are applicable across subfield boundaries, for example, when we think of more natural science fields such as nutrition science and more social science fields such as the sociology of health. One previously published reporting guideline is based only on studies in palliative care and their primary objective of consensus building [
]. Subfield-specific differences can also be observed in other parameters of the Delphi process, for example, determining the level of consensus. For developing medical guidelines, the level of consensus is defined [
]. In the health sciences, consensus is defined primarily in terms of percentage agreement but also by central tendency, formal measure of agreement (e.g. Crohnbach’s alpha, Kendal’s W) or a combination of several measures [
In the health sciences in general there is a broad variation regard to the use and valuation of methods (qualitative or quantitative) and in the analysis of data (such as statistical tests in relation to the scale length, procedure for analyzing responses of open-ended question) [
]. These different research approaches also differ in their epistemological grounding, as they are anchored in different theories of science and knowledge cultures (e.g., natural science versus social science) [
Delphi-Verfahren in den Gesundheitswissenschaften – erkenntnistheoretische Potenziale und Herausforderungen.
in: Niederberger M. Renn O. Delphi-Verfahren in den Sozial- und Gesundheitswissenschaften: Konzept. Varianten und Anwendungsbeispiele,
Springer VS, Wiesbaden2019: 55-81
]. This also holds true for the Delphi technique. In general, the Delphi technique is rooted in different epistemological traditions and these epistemological groundings also vary with the specific type of Delphi that is applied [
Delphi-Verfahren in den Gesundheitswissenschaften – erkenntnistheoretische Potenziale und Herausforderungen.
in: Niederberger M. Renn O. Delphi-Verfahren in den Sozial- und Gesundheitswissenschaften: Konzept. Varianten und Anwendungsbeispiele,
Springer VS, Wiesbaden2019: 55-81
]. Thus, a reporting guideline must take this diversity into account. The aim of this study is to identify the similarities, differences, and possible shortcomings of existing reporting guidelines, particularly regarding epistemological and methodological aspects, and to draft an initial proposal for a comprehensively applicable reporting guideline in the health science sector.
Methodical approach
We conducted a methodological review to analyze existing reporting guidelines on Delphi techniques in the health sciences. As no reporting guideline has yet been published on this, we have used the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analysis) statement as guidance for reporting [
]. The review was not pre-registered and no study protocol is available. For our methodological review it was not possible to apply all of the topics and proposed analyses included in the PRISMA statement (prior definition of data items, study risk of bias assessment, effect measures, quantitative synthesis methods und results (e.g., sensitivity or subgroup analyses, summary measures, meta-regression), certainty assessment, registration and protocol) mainly due to the examined datatypes, the inductive approach, and the accuracy of fit of the guideline to the to the methodological focus of the study.
Literature search - Identification of the reporting guidelines
The identification of existing reporting guidelines on Delphi techniques in the health sciences was done in five different ways. Table 1 gives an overview.
] of reporting practices for Delphi studies, two of the authors (MN, JS) identified systematic reviews of Delphi studies in the health sciences in 2019. For the current study, the reviews included in the overview (n = 12) were searched for reporting guidelines by two of the authors (MN, JS) independently using full text screening. Reviews without recommendations on reporting were excluded.
2.
One author (JS) performed a supplemental search. The initial search yielded 139 hits, which were screened by one author (JS) based on titles and abstracts for a methodological focus of the studies and recommendations on reporting Delphi studies. Primary studies that seemed to report only results of their own Delphi process were excluded. The full texts of those considered potentially relevant were screened by one author (JS) for reporting guidelines on Delphi studies.
3.
On April 21, 2021, the scientific Delphi expert network (https://delphi.ph-sg.de/) conducted a literature search on Delphi studies in central data resources for the social and health sciences (Scopus, MEDLINE, CINAHL, and Epistemonikos). The German-speaking scientific network DeWiss consists of 20 researchers from different subfields and epistemological backgrounds and focuses on epistemological foundations, methodological issues and guidelines for the implementation and reporting of Delphi studies. The aim of the literature search was to create two data resources on 1) Delphi primary studies and 2) methodological studies on Delphi techniques (available via https://www.zotero.org/groups/4396781/dewiss_datenbanken_delphi-studien). The inclusion criteria for building the data resources are briefly described below. Original papers with Delphi studies were assigned to the two data resources via title-abstract screening by six researchers (MN, SD, JS) and research fellows (MT, LS, XS) of the Delphi network without verification by a second reviewer. A total of 155 articles were assigned to the database with methodological studies (2) and 7,044 articles to the data resource of Delphi primary studies (1). In the current study, a sub-analysis was performed in the two data resources. The data resource of methodological studies (2) was searched via title-abstract-screening for recommendations for reporting Delphi studies by a research fellow (MT) and the full texts of those considered potentially relevant were screened by one of the authors (JS). The filtered data resource of Delphi primary studies (1) (see Table 1) yielded in n = 1,683 articles. The title, abstract and the full texts of those considered potentially relevant were screened for reporting recommendations by one of the authors (JS) and a research fellow (XS), not independently. Studies that seemed to report only the results of their own Delphi process were excluded.
4.
In addition, the authors performed a search of the EQUATOR network data resource. One author (JS) screened the hits for recommendations on reporting Delphi studies.
5.
We also included articles in the screening process, which had come to attention by members of the Delphi expert network (DeWiss) in the course of their own research or through scientific networks. All of the authors screened the full texts of the proposed articles for recommendations on reporting Delphi studies.
Table 1Overview of data resources and search strategies.
No.
Data resources
Search strategy
1
Overview (previous research) conducted in PubMed in April 2019
Keywords: “review” and “delphi”, English or German language, no time limit.
2
Supplementary search of the overview (see No. 1) in PubMed conducted in June 2021
Keywords: (Delphi[Title]) AND ((Guideline[Title]) OR (Guidelines[Title]) OR (Reporting[Title]))English or German language, no time limit.
3
Sub-analysis of the Delphi expert network (DeWiss) data resources on 1) Delphi primary studies and 2) methodological studiesconducted in Scopus, MEDLINE, CINAHL, Epistemonikosin April 2021
1) primary studies: keywords “delphi*”, “reporting” and “guideline” in title or abstract, English or German language, from 2016 to April 2021. 2) methodological studies: keyword “delphi*” in title or abstract, English or German language, from 2016 to April 2021.
4
EQUATOR network search conducted in April 2021
Keyword: Delphi
5
Contacting experts in January 2022
Manual search of the Delphi expert network (DeWiss) members in the course of their own research
]. In doing so, we pursued two objectives: 1) to provide an overview of the range of dimensions in the reporting guidelines; 2) to reveal the frequencies of the dimensions mentioned. Here, content analysis is suitable for both qualitative and quantitative evaluation and presentation of results [
]. In a first step, two researchers (JS, AH) independently paraphrased, summarized and structured all relevant text passages close to the original text. They formed paraphrases from the text material and inductively determined categories from them. The independent procedure was intended to achieve as much intersubjectivity as possible [
]. In a second step, a third researcher (MN) compared the category systems of the two researchers and in the case of divergent coding, moderated the exchange between the researchers. The goal was to consensually merge both category systems into a unified system. In a third step, all of the paraphrases were reviewed again and classified in the category system. It was also noted to which publications the respective paraphrases could be assigned. It became apparent that some paraphrases could not be clearly assigned to a specific category. In these cases, the research team (JS, AH, MN) discursively agreed on an assignment. For the presentation of the frequency, we determined the number of publications that contained at least one statement (paraphrase) on the respective subcategory (dimension).
Proposal for a Delphi reporting guideline
Based on the synthesis of existing reporting guidelines and against the background of the Delphi expert network’s discussion about the epistemological foundations of the Delphi technique, we drafted an initial proposal for Delphi reporting guidelines that is applicable regardless of variant, objective or subfield and reflected on possible limitations of the analyzed recommendations on reporting Delphi studies. Scientists from the Delphi expert network reviewed and consented to this proposal and reflected on its coverage for subfields in the health sciences.
Results
Literature search
We identified a total of ten articles with reporting guidelines for Delphi studies in the health science sector (see Table 2). Figure 1 gives an overview of the results of the literature search.
Table 2Overview of reporting guidelines for Delphi procedures.
ID
Authors
Year
Journal
Topic
Methodology
Research goal
1
Boulkedid et al.
2011
PLOS ONE
Health services
Systematic review including 80 Delphi studies
Describe how the selection of health quality indicators should be reported in Delphi studies
2
Diamond et al.
2014
Journal of Clinical Epidemiology
Not limited
Systematic review, random sample of 100 Delphi studies
Describe how to clearly document consensus in Delphi studies
3
Hasson et al.
2000
Journal of Advanced Nursing
Nursing profession
Theoretical considerations of 3 experts (authors)
Provide health professionals with an understanding of the preparation, action steps, and difficulties of Delphi studies
4
Humphrey-Murto et al.
2017
Academic Medicine
Medical education research
Theoretical considerations of authors with examples from the relevant research literature.
Suggestions (“how to use guide”) that researchers should consider when planning and publishing studies using consensus group methods
5
Humphrey-Murto et al.
2019
Journal of Rheumatology
Clinical trials in rheumatic diseases
Evaluation of a preliminary developed checklist by 170 experts (clinicians, researchers, patients, pharmaceutical representatives, fellows and regulatory authorities)
Provide a guide on how to clearly outline the multiple aspects of the Delphi process
6
Iglesias et al.
2016
Pharmaco Economics
Model-Based economic evaluations of healthcare interventions
Two-round online Delphi process with 15 experts with previous experience on studies to identify expert judgments in the context of economic evaluations in healthcare
Produce guidelines for reporting expert elicitation studies and Delphi studies to capture expert opinion
7
Jünger et al.
2017
Journal of Palliative Medicine
Palliative care
Systematic review including 30 Delphi studies
Develop best practice guidelines for reporting Delphi studies
8
Sinah et al.
2011
PLOS Medicine
Clinical trials
Systematic review including 15 Delphi studies
Develop a core set of outcomes for presenting measurement in clinical trials
9
Toma & Picioreanu
2016
International Journal of Public Health Research
Primary care
Theoretical considerations of authors with examples from the relevant research literature.
Describe the methodological steps, identify systematic bias, and propose a reporting and reading template for Delphi studies.
10
Waggoner et al.
2016
Academic Medicine
Not limited
Literature review including 50 studies using consensus methodology
Guidelines for describing the use of a consensus methodology (Delphi, nominal group process or consensus development panel)
] contained specific recommendations for reporting Delphi results. The supplemental search enabled the inclusion of one additional study. Four articles could be included in the search within the two data resources of the Delphi expert network. No further hits were detected by searching the EQUATOR network. Table 2 gives an overview of all publications that contain explicitly formulated guidelines for reporting. Studies that report only the results of their own Delphi process and not recommendations on reporting Delphi studies were excluded (e.g. [
We inductively formed nine main categories: Justification, Expert panel, Questionnaire, Survey design, Process regulation, Analyses, Results, Discussion, Methods reflection & Ethics. Each of the main categories had two to five subcategories. The paraphrasing shows that the range of aspects within the subcategories differs considerably.
The frequencies show that some subcategories are addressed only by single reporting guidelines (ID number indicates order in which the publication is listed in Table 2): meeting of participants (ID1), proceeding with the survey method (ID7), transfer of the results (ID7), validation (ID7), prevention of bias (ID7). Only the subcategory “consensus” is mentioned in all ten publications examined. Most of the reporting guidelines refer to the study objective (n = 6 (ID1, ID3, ID4, ID6, ID7, ID9)), the expert panel (expert selection n = 9 (all except ID1) and panel size and composition n = 9 (all except ID2)), the development of the questionnaire (n = 7 (ID1, ID4-ID9)), the number and design of rounds (n = 7 (ID1, ID3-ID7, ID10)), the feedback design (n = 7 (ID1, ID4-ID9)), the response rate (n = 6 (ID3, ID4, ID6-ID9)), and the form in which the results are presented (n = 7 (ID1, ID3, ID4, ID6-ID9)). Reflections on the epistemological foundations of the Delphi technique, however, are only mentioned in one publication (ID7) and even there with only a minor reference to the constructivist character of Delphi techniques (see Table 3).
Refers to the number of listed publications (Table 2) in which this subcategory was mentioned;
Paraphrases
I Justification The reporting guidelines require disclosure of the research question or the objective of the Delphi techniques, the question of how the evidence base was taken into account, and the justification for choosing the Delphi technique.
Rationale
5
-
Rationale for the choice of Delphi (ID3,ID4,ID5,ID6,ID7)
ID number indicates order in which the publication is listed in Table 2.
-
Topic justification (ID3,ID6)
-
Keeping Delphi’s constructivist nature in mind (ID7)
Study objective
6
-
Aim/purpose of the study (ID1,ID4,ID7,ID9)
-
Research problem clearly defined (ID3,ID6)
-
Explicit research question (ID6)
-
Suitability of study objective and Delphi (ID4,ID6,ID7)
Explanation of the Delphi method
5
-
Methodological objective (ID2)
-
Clear explanation of the method (ID3,ID6)
-
Justification for any modification to the Delphi technique (ID4,ID7)
Evidence on the topic
4
-
Literature review (ID3,ID4,ID6)
-
Topic under study (ID3)
-
Evidence base for the topic (ID6,ID7)
II Expert panel It is recommended to report on the selection of the experts, the composition, the sample size, the approach, the handling of anonymity and necessary background information.
Expert selection
9
-
Selection criteria (ID2,ID4,ID7,ID6)
-
Selection/sampling process (ID3,ID8,ID9)
-
Rationale for inclusion/exclusion (ID2,ID4,ID5,ID9)
-
Definition of expert (ID4,ID6,ID9,ID10)
Expert recruitment
5
-
Geographic scope (ID1)
-
Methods of recruitment (ID1,ID6,ID7,ID8,ID10)
Anonymity
4
-
Level of anonymity (ID8)
-
Maintaining anonymity (ID4,ID5)
-
Acknowledgement of anonymity (ID6)
Panel size and composition
9
-
Expert characteristics (ID1,ID3,ID4)
-
Panel composition (ID1,ID6,ID8,ID10)
-
Number of participants, panel size (ID5,ID8,ID10)
-
Method for determining the sample size (ID5)
-
Number of groups in each step (ID4,ID5)
-
Sociodemographic details (ID7)
-
Information on expertise (ID7,ID9)
-
Types of participants (ID8)
-
Willingness to participate (ID9)
Provided materials
5
-
Background information provided (if necessary, per group of participants) (ID4,ID5,ID9)
-
Information/material provided before the first round (ID7,ID8)
III Questionnaire It is recommended to describe the survey instrument; this concerns the structure, the scope, and the research questions on which basis the items were formulated.
Questionnaire development
7
-
Item generation for the first questionnaire (ID1,ID4,ID5,ID8)
Number of items in the first questionnaire (ID1,ID9)
-
Number of issues generated (ID3)
-
What was asked in each round? (ID8)
IV Survey design Indication of the number of rounds and presentation of the feedback design are recommended, partly also the duration of the Delphi procedure and the termination criterion.
Rounds
7
-
Number of rounds (ID1,ID3,ID4,ID6,ID7)
-
Outline of each round (ID3)
-
A priori decision of round number? (ID5)
-
Plans for moving from one round to the next (ID6)
-
Explanation of reason for more than two rounds (ID10)
-
Design of rounds (ID7)
Meeting of participants
1
-
Was a meeting held? (ID1)
-
Description of meeting participants (ID1)
Feedback design
7
-
Mode of feedback to experts after each round (ID1,ID4,ID5,ID6,ID8)
-
Quantitative and qualitative feedback (each round) and overall group response (ID9)
-
Is feedback given to the group and/or individual? (ID6)
-
Processing and synthesis of experts’ responses to inform the subsequent round (ID7)
Stopping criterion
5
-
Stopping criteria (in absence of consensus) (ID2,ID4,ID9)
-
Threshold to terminate the process (ID2,ID7)
-
Terminating criteria when not determined a priori (ID5)
Duration
2
-
Duration of the Delphi procedure (ID1,ID9)
V Process regulation In part, criteria are formulated for the survey mode and the development of the items. Single reporting guidelines list the expert panel's reflections on the various Delphi rounds, especially from the perspective of dissent, as well as information on a pretest.
Pretest
2
-
Piloting of material and survey instruments (ID4,ID7)
-
Reviewing and piloting background information in advance (ID7)
Flow of items eliminated and added in each round (ID1)
-
Criteria for dropping items (ID2)
-
Criteria to include terms/items in subsequent rounds (ID5,ID9)
-
Proceeding with items or topics in the next round (ID7)
Proceeding with non-responders
3
-
Invitation/exclusion of non-responders in subsequent rounds (ID5,ID8,ID9)
-
Management of non-responders (ID5)
Proceeding with the survey method
1
-
Modification of the survey instrument and methodological decisions by the research team based on previous rounds (ID7)
VI Analyses All reporting guidelines call for the definition and handling of consensus.
Methods of data analysis
5
-
Method justification (ID3)
-
Management of (minority) opinions (ID6,ID9)
-
Analyzing and handling qualitative and quantitative data (ID6,ID9)
-
Preparation of a pre-specified data analysis plan (ID6)
-
Methods of data analysis (ID7)
-
Statistical analysis for each question (ID10)
Definition and dealing with consensus
10
-
Definition of consensus (ID1,ID2,ID4,ID5,ID7,ID8,ID9)
-
Issues of consensus (ID3)
-
Interpretation of consensus (ID3)
-
Description of the meaning of consensus in relation to the study aim (ID6)
-
Definition of agreement (ID6)
-
A priori definition (ID7,ID8)
-
Procedure when consensus is reached (ID7)
-
Explanation of what constituted consensus (ID10)
-
Explanation of how consensus was assessed (ID10)
Dealing with dissent
3
-
Interpretation of consensus not reached (ID3)
-
Are outliers to the overall consensus asked for the reasons for their answers? (ID6)
-
Procedures when consensus is not reached (ID7)
-
Strategies to deal with non-consensus (ID7)
VII Results The reporting guidelines require the disclosure of response rates and the presentation of results differentiated according to the various Delphi rounds. The presentation of the methodological procedure in graphic form, e.g., a flowchart, is also recommended.
Response rate
6
-
Response rates in each round (if necessary, per group of participants) (ID3,ID4,ID6,ID7,ID8,ID9)
-
Non-response rate (ID7)
-
Number who completed every round (ID8)
Reporting results
7
-
List of selected items (ID1)
-
List of all outcome participants agreed/selected (ID1,ID8,ID9)
-
Results of each score (ID1)
-
Results for each round (ID3,ID4,ID7,ID8)
-
Presentation of results in round 2, indicating the strength of support in round (ID3)
-
Summary of the total number of issues generated in round 1 (ID6)
-
Summary of the strength of overall consensus in the final round (ID6)
-
Graphic and/or statistical representations (ID6)
-
Figures showing the average group response (ID7)
-
Changes between rounds (ID7)
-
Distribution of each outcome in the final round (ID8)
Reporting the overall process
4
-
Diagram of participant flow (ID1)
-
Outline of the overall process, description of each step (ID5)
-
Flow chart for illustration, including preparatory phase, rounds, participants, interim and concluding steps (ID4,ID7)
VIII Discussion Data are interpreted and other uses discussed; dissemination of the results and further research needs should be identified.
Interpretation of results
3
-
Statistical interpretation (ID3)
-
Interpretation guideline for the reader (ID3,ID6)
-
Presentation of the meaning of the results (ID6)
-
How to digest the findings in relation to the emphasis being placed upon them (ID6)
-
Interpretation of consensus (not necessarily imply the correct answer) and non-consensus (provide informative insights and highlight differences in perspectives) (ID7)
Transfer of the results
1
-
Clearly identifiable resulting guidance (ID7)
-
Recommendations for transfer into practice and implementation (ID7)
-
Reference to a more detailed presentation (ID7)
Dissemination plan
2
-
Dissemination plan to facilitate implementation (ID7)
-
Were the final results of the study sent to the panel experts? (ID9)
Further research
2
-
Direction of future research (ID3,ID6)
IX Methods reflection & Ethics The results are to be discussed with regard to the quality of the data and possible limitations or biases. In some reporting guidelines it is recommended to disclose in the report how the anonymity of the respondents was handled and whether ethical aspects were taken into account in the survey process.
Validation
1
-
Review of the final draft by an external board or authority before publication (ID7)
Validity/Reliability
2
-
Identification of reliability and validity issues (ID3,ID6)
Discussion of limitations
4
-
Discussion of methodological issues (ID4,ID5)
-
Recognition of the influence of the composition of the panel on the results (ID6)
-
Limitation of reporting data from quantitative questions (ID6)
-
Critical reflection on potential limitations and their impact (ID4,ID7)
-
Discussion of lack of consensus to inform the future research agenda (ID4)
-
Adequacy of conclusions (ID7)
Prevention of bias
1
-
Examining the effect of background information on expert judgment (ID7)
-
Measures to avoid directly or indirectly influencing experts’ judgments (ID7)
Further ethical issues
2
-
Ethical responsibilities/issues towards experts, sample and research community (ID3,ID6)
1 Refers to the number of listed publications (Table 2) in which this subcategory was mentioned;
2 ID number indicates order in which the publication is listed in Table 2.
The challenge in preparing a comprehensive Delphi reporting guideline is to consider the diversity of Delphi variants and the respective modifications of the Delphi technique. Involving other scientists from the Delphi expert network, we formulated an initial proposal for Delphi reporting guidelines in Figure 2 based on an epistemological discussion and synthesis of existing reporting guidelines. These criteria appear central to reporting on Delphi studies regardless of variant, objective or subfield.
Figure 2Proposal for reporting guidelines on Delphi techniques in the health sciences.
The reporting guidelines analyzed here relate to Delphi studies in health sciences. None of the identified reporting guidelines claim to cover the entire Delphi process or the spectrum of Delphi variants or their possible applications in the different health science subfields. As a consequence, the criteria are divergent and only the aspect of consensus appears in all of the reporting guidelines. In some cases, the information in the reporting guidelines is imprecisely formulated because it is not specified whether the information refers to the beginning of the Delphi technique, the process or the results. Especially information about the expert panel or the questionnaire can change during the Delphi study. Sometimes it remains unclear when methodological decisions have to be made or defined. This particularly concerns the consensus definition, which can be made a priori or a posteriori.
The reporting guidelines examined were developed on the basis of Delphi studies that followed more of a traditional approach, such as maintaining anonymity or multiple completed rounds. Hence, the variety of Delphi variants described in the introductory section is not sufficiently considered. This is critical because some Delphi variants break up core characteristics of Delphi techniques, such as anonymity in a group Delphi or the round-based principle in the real-time Delphi. This means that in some cases other aspects have to be reported for different Delphi variants [
], basic methodological characteristics of the Delphi process, such as iterative rounds and the receiving of feedback, are not always given in studies labeled as Delphi technique. In relation to the research goal, Delphi studies are conducted in healthcare particularly to identify consensus [
]. A comprehensive reporting guideline could make Delphi studies with different subfield-specific objectives comparable, including in terms of their quality. Likewise, it can highlight the variety of possible applications and complexity of the process. A Delphi process includes several steps of preparing, conducting and analyzing and is thus also more complex and time-consuming than single-stage survey procedures. Thus, a minimum of 16 weeks is recommended for the Delphi process alone, with an additional six to ten weeks for writing a scientific manuscript [
Also, the discussion about the epistemological foundations of the Delphi technique in general and of the specific, applied Delphi variant in particular is often neglected and is not adequately considered in the existing reporting guidelines (see Table 3). One reason could be that the practical implications cannot be derived directly. However, the question of whether the Delphi technique is based on a realist perspective following critical rationalism or on a constructivist perspective is relevant for the evaluation of the quality of a Delphi process and its results. We assume that critical rationalism is more prevalent in the natural sciences and constructivism is more prevalent in the social sciences. This means:
], a Delphi technique should be able to produce an approximation of “true” or objective knowledge by correcting the possibly biased expert judgments through feedback from the respondent group. Thus, a consensus among the participants can be interpreted as an approximation of “true” knowledge.
] showed in a methodological study on the Delphi technique that different stakeholder panels in primary mental health care “have diverse views of quality of care and these differences translate into how they rate quality indicators” [
]. From a constructivist perspective, this comes not as a surprise and would also not be regarded as a problem. Form a critical rationalist perspective, however, this can be as rendered as failure with regard to establishing consensus and approximating a “true” value. Thus, depending on the epistemological perspective, one can arrive at opposite conclusions concerning the quality of the respective Delphi results.
The bottom line is that reporting guidelines encourage researchers to reflect on and disclose key assumptions and methodological choices, as well as the epistemological foundations on which basis results are interpreted. The criteria of the newly developed proposal for Delphi reporting guidelines (see Fig. 2) are not to be seen as completely independent of each other. From a methodological perspective they are closely interrelated and must form a coherent frame. This concerns, in particular, the fit of the epistemological positioning, the weighting of qualitative or quantitative aspects, and the quality criteria. Concerning all criteria, it is also important to disclose the time of definition or determination during the Delphi techniques. Furthermore, all criteria should be reflected throughout the individual Delphi rounds.
Limitations
There are also at least three potential limitations concerning this study, which should be mentioned here. The screening and inclusion of articles was partly based on the decision of a single researcher. For resource reasons, an independent assessment of multiple researchers, as it was applied for the content analysis, was not possible at all steps. The inclusion of further recommendations from the Delphi expert network should take this limitation into account and draw attention to possibly missed key articles in the research field. Furthermore, this proposal for a Delphi reporting guideline cannot be assumed in terms of international representativeness. Neither the analyzed guidelines nor the Delphi expert network could fulfill these requirements. However, the group of authors as well as the Delphi expert network consists of experts from different subfields and ranges of experience in the application of the Delphi technique (e.g., concerning different Delphi variants). On this point, the heterogeneity of expertise was considered. Finally, the reporting and conducting of the study could have been improved with an appropriate reporting guideline for methodological reviews and an a priori published study protocol.
Conclusion
The methodological review provides an overview of existing reporting guidelines for Delphi studies in the health sciences. The content analysis shows that they vary widely, both in terms of criteria and level of detail. Most reporting guidelines explicitly address reporting on consensus definition, expert panel, response, feedback design, and number of rounds. However, they lack differentiation, for example, with regard to different Delphi variants and objectives. In particular, epistemological principles have been neglected so far, even though they are central to assessing the quality of the procedure and classifying the collected data. Based on a synthesis of existing reporting guidelines and in discussion with experts from different subfields with particular consideration of epistemological questions, we have developed an initial proposal for a comprehensive reporting guideline for the health sciences. The results of our methodological review do not provide a final reporting guideline, but it is an important step toward improving the quality of Delphi techniques. The different criteria need to be further differentiated (in an expert discourse).
Funding
The DeWiss Network is supported by the German research foundation (Project Number: 429572724). The study is part of the project DELPHISTAR – Delphi studies in social and health sciences – recommendations for a standardized reporting. The project has been registered at the Equator network platform and the OSF platform and can be assessed at this page: https://osf.io/gc4jk.
Acknowledgements
For their support in the screening process we would like to thank Stefanie Deckert, Leonie Salzmann, Xenia Sestak and Michaela Trompke.
Conflict of interest
The authors declare that there are no conflicts of interest.
CRediT author statement
Marlen Niederberger: Conceptualization, analysis of the reporting guidelines, writing (original draft) and corrections.
Julia Spranger: Research and analysis of the reporting guidelines, writing (original draft) and corrections.
Angelika Homberg: Analysis of the reporting guidelines, writing (original draft) and corrections.
Marco Sonnberger: Writing (original draft) and corrections.
Methodischer Standard für die Entwicklung von Qualitätsindikatoren im Rahmen von S3-Leitlinien – Ergebnisse einer strukturierten Konsensfindung (Methodological Standard for the Development of Quality Indicators within Clinical Practice Guidelines – Results of a structured consensus process).
White Paper for global palliative care advocacy: recommendations from a PAL-LIFE expert advisory group of the Pontifical Academy for Life, Vatican City.
A national consensus management pathway for paediatric inflammatory multisystem syndrome temporally associated with COVID-19 (PIMS-TS): results of a national Delphi process.
Using the Delphi technique to determine which outcomes to measure in clinical trials: Recommendations for the future based on a systematic review of existing studies.
Delphi-Verfahren in den Gesundheitswissenschaften – erkenntnistheoretische Potenziale und Herausforderungen.
in: Niederberger M. Renn O. Delphi-Verfahren in den Sozial- und Gesundheitswissenschaften: Konzept. Varianten und Anwendungsbeispiele,
Springer VS, Wiesbaden2019: 55-81
00076-9&id=gr1.jpg){kind=link}
00076-9&id=gr2.jpg){kind=link}