Advertisement

Reporting guidelines for Delphi techniques in health sciences: A methodological review

Open AccessPublished:June 16, 2022DOI:https://doi.org/10.1016/j.zefq.2022.04.025

      Abstract

      Background

      Delphi techniques are conducted across different subfields in the health sciences. The reporting practices of studies using Delphi techniques vary, and current reporting guidelines for Delphi techniques focus on individual subfields of the health sciences or on different aspects of research and are therefore of limited applicability. The aim of this article was to identify similarities, differences, and possible shortcomings of existing Delphi reporting guidelines and to draft an initial proposal for a comprehensively applicable reporting guideline.

      Methods

      A systematic literature search for reporting guidelines on Delphi studies was performed in existing data resources based on databases in the health sciences (Scopus, MEDLINE, CINAHL, Epistemonikos) including publications from 2016 to 2021. In June 2021, we conducted an additional search in PubMed and included further studies by contacting experts of the scientific Delphi expert network (DeWiss). Title and abstract screening of articles was performed, followed by a full-text screening of the articles included. We qualitatively and quantitatively evaluated, compared and contrasted the reporting guidelines identified using content analysis and discussed the results among the members of the Delphi expert network.

      Results

      We retrieved ten health science articles with reporting guidelines for Delphi studies. In analyzing them, we identified nine main categories (Justification, Expert panel, Questionnaire, Survey design, Process regulation, Analyses, Results, Discussion, Methods reflection & Ethics). The current reporting guidelines vary significantly, with only the aspect of consensus appearing in all of them. Frequency distributions show that most of the subcategories are only addressed in individual articles (e. g., meeting of participants, proceeding with the survey method, transfer of the results, validation, prevention of bias) and that epistemological foundations of the Delphi technique are rarely mentioned or reflected on. We drafted an initial proposal for Delphi reporting guidelines for the health science sector.

      Discussion

      A well-justified position concerning epistemological foundations of Delphi studies is necessary to make the quality of the process assessable and, along with the reporting of the process, to classify and compare study results. This will increase the acceptance of both the method in the health science sector and the results in medical practice. A Delphi reporting guideline must, above all, take into account the diversity of variants, subfield-related objectives and application areas, and their modifications of the Delphi technique in order to be comprehensively applicable in the health sciences.

      Conclusion

      The results of our methodological review do not provide a final reporting guideline. The newly developed proposal is intended to encourage discussion and agreement in further analyses.

      Zusammenfassung

      Hintergrund

      Delphi-Verfahren werden in den Gesundheitswissenschaften disziplinübergreifend durchgeführt. Bisher ist die Berichterstattung von Studien mit Delphi-Verfahren sehr unterschiedlich, und die vorhandenen Reporting Guidelines zu Delphi-Verfahren weisen unterschiedliche gesundheitswissenschaftliche teilbereichs- oder forschungsspezifische Schwerpunktsetzungen auf und sind daher nur eingeschränkt anwendbar. Ziel dieses Beitrags ist es, Gemeinsamkeiten, Unterschiede sowie etwaige Mängel bestehender Reporting Guidelines für Delphi-Studien herauszuarbeiten und einen ersten Vorschlag für eine umfassend anwendbare Delphi-Reporting-Guideline zu formulieren.

      Methode

      Wir führten eine systematische Literaturrecherche nach Reporting Guidelines zu Delphi-Studien in zwei Datenbeständen mit Publikationen zu Delphi-Studien aus den Jahren 2016 bis April 2021 aus Datenbanken im gesundheitswissenschaftlichen Bereich (Scopus, MEDLINE, CINAHL, Epistemonikos) durch. Ergänzend erfolgte eine Suche in PubMed im Juni 2021 und weitere Studien wurden über die Kontaktierung des wissenschaftlichen Delphi Expert*innennetzwerks (DeWiss) eingeschlossen. Das Screening der Artikel erfolgte anhand von Titel und Abstract, gefolgt von einem Volltext-Screening der eingeschlossenen Artikel. Die identifizierten Reporting Guidelines wurden mithilfe einer Inhaltsanalyse qualitativ und quantitativ ausgewertet, gegenübergestellt und verglichen und im Autor*innenteam und mit weiteren Expert*innen des Delphi-Netzwerks diskutiert.

      Ergebnisse

      Es wurden zehn Artikel mit Reporting Guidelines für Delphi-Studien identifiziert. Durch die Inhaltsanalyse konnten neun Hauptkategorien (Begründung, Expertenpanel, Fragebogen, Erhebungsdesign, Prozessgestaltung, Analysen, Ergebnisse, Diskussion, Methodenreflexion & Ethik) gebildet werden. Die vorliegenden Berichterstattungsleitlinien unterscheiden sich erheblich und nur der Aspekt der Konsensfindung wird in allen genannt. Die quantitative Analyse zeigt, dass einzelne Subkategorien nur in einzelnen Artikeln adressiert (bspw. Treffen der Teilnehmer*innen, Umfrageverlauf, Ergebnistransfer, Validierung, Prävention von Verzerrungen) und epistemologische Grundlagen der Delphi-Methode kaum angesprochen und reflektiert werden. Ein erster Vorschlag für eine Delphi-Reporting-Guideline im Gesundheitsbereich wurde formuliert.

      Diskussion

      Eine Positionierung im Hinblick auf epistemologische Fragen ist notwendig, um die Prozessqualität bewertbar zu machen und mithilfe der Berichterstattung über den Delphi-Prozess Studienergebnisse vergleichen und einordnen zu können. Dies trägt zur Akzeptanz der Methode in den Gesundheitswissenschaften und von deren Ergebnissen in der Praxis im Gesundheitswesen bei. Darüber hinaus muss eine Delphi-Berichterstattungsleitlinie die Vielfalt der Varianten, teilbereichsspezifischen Ziele und Anwendungsbereiche sowie die jeweiligen Modifikationen der Delphi-Technik berücksichtigen, um eine umfassende Anwendung in den Gesundheitswissenschaften zu ermöglichen.

      Schlussfolgerung

      Die Ergebnisse unseres methodischen Reviews stellen keine endgültige Richtlinie für die Berichterstattung dar. Der neu entwickelte Vorschlag sollte in weiteren Analysen reflektiert und konsentiert werden.

      Keywords

      Schlüsselwörter

      Introduction

      Delphi techniques are group discussion procedures in which complex issues about which uncertain and incomplete knowledge exists are assessed by experts in an iterative and structured process [
      • Linstone H.A.
      • Turoff M.
      The delphi method.
      ,
      • Humphrey-Murto S.
      • Varpio L.
      • Wood T.J.
      • Gonsalves C.
      • Ufholz L.-A.
      • Mascioli K.
      • Wang C.
      • Foth T.
      The use of the Delphi and other consensus group methods in medical education research: a review.
      ,
      • Hutchings A.
      • Raine R.
      A systematic review of factors affecting the judgments produced by formal consensus development methods in health care.
      ,
      • Niederberger M.
      • Köberich S.
      members of the DeWiss Network, Coming to consensus: The Delphi technique.
      ]. They are conducted in different subfields of health science (e.g., clinical patient care, public health and health promotion, health service research, medical education), particularly to identify consensus [
      • Niederberger M.
      • Spranger J.
      Delphi technique in health sciences: A Map, Front.
      ,
      • Jorm A.F.
      Using the Delphi expert consensus method in mental health research.
      ,
      • Nasa P.
      • Jain R.
      • Juneja D.
      Delphi methodology in healthcare research: How to decide its appropriateness.
      ]. Evidence-based knowledge, practical knowledge, and sometimes everyday knowledge based on life-world experiences are brought together in a structured process. The goal is to develop consensual guidelines or standards that are then used to support decision-making and action in the health sector. The methodological literature describes five characteristics of Delphi techniques [
      • Niederberger M.
      • Köberich S.
      members of the DeWiss Network, Coming to consensus: The Delphi technique.
      ,
      • Diamond I.R.
      • Grant R.C.
      • Feldman B.M.
      • Pencharz P.B.
      • Ling S.C.
      • Moore A.M.
      • Wales P.W.
      Defining consensus: A systematic review recommends methodologic criteria for reporting of Delphi studies.
      ,
      • Hasson F.
      • Keeney S.
      • McKenna H.
      Research guidelines for the Delphi survey technique.
      ]:
      • 1.
        Experts with specific knowledge are interviewed, usually while preserving their anonymity.
      • 2.
        The survey is conducted in at least two Delphi rounds, more than three Delphi rounds being the exception.
      • 3.
        The survey instrument is usually a standardized questionnaire, often with open-ended questions to capture arguments and reasoning horizons.
      • 4.
        The statistical analysis is usually based on descriptive calculations.
      • 5.
        From the second Delphi round onwards, the experts receive feedback on the results of the previous round in addition to the questionnaire and can reconsider or revise their judgments.
      There are other Delphi variants in addition to the traditional Delphi approach. The most important variants are the following:
      • A real-time Delphi in which expert judgments are reflected back online and in real time; i.e., respondents can make their judgments several times, and there are no separate individual Delphi rounds [
        • Gnatzy T.
        • Warth J.
        • von der Gracht H.
        • Darkow I.-L.
        Validating an innovative real-time Delphi approach – A methodological comparison between real-time and conventional Delphi studies.
        ].
      • A policy Delphi is concerned with explicitly capturing a wide range of judgments or approaches toward innovations or solutions [
        • Franklin K.K.
        • Hart J.K.
        Idea generation and exploration: benefits and limitations of the Policy Delphi research method.
        ].
      • An argumentative Delphi in which the focus is on the qualitative reasoning of the standardized judgments [
        • Seker S.E.
        Computerized Argument Delphi technique.
        ]. These are now mostly performed as a dynamic argumentative Delphi (DAD) in real-time.
      • A group Delphi to which the experts are invited to openly discuss arguments for divergent judgments. This removes the anonymity of the experts [
        • Niederberger M.
        • Köberich S.
        members of the DeWiss Network, Coming to consensus: The Delphi technique.
        ].
      • A fuzzy Delphi utilizes triangulation statistics to determine the distance between judgments within the expert panel [
        • Habibi A.
        • Jahantigh F.F.
        • Sarafrazi A.
        Fuzzy Delphi technique for forecasting and screening items, Asia.
        ].
      Through a Delphi process, diverging expert assessments and opinions become transparent and ultimately–at least this is the claim–resolved and consented upon. This applies in particular to clinical guidelines [
      • Deckert S.
      • Arnold K.
      • Becker M.
      • Geraedts M.
      • Brombach M.
      • Breuing J.
      • Bolster M.
      • Assion C.
      • Birkner N.
      • Buchholz E.
      • Carl E.-G.
      • Diel F.
      • Döbler K.
      • Follmann M.
      • Harfst T.
      • Klinkhammer-Schalke M.
      • Kopp I.
      • Lebert B.
      • Lühmann D.
      • Meiling C.
      • Niehues T.
      • Petzold T.
      • Schorr S.
      • Tholen R.
      • Wesselmann S.
      • Voigt K.
      • Willms G.
      • Neugebauer E.
      • Pieper D.
      • Nothacker M.
      • Schmitt J.
      Methodischer Standard für die Entwicklung von Qualitätsindikatoren im Rahmen von S3-Leitlinien – Ergebnisse einer strukturierten Konsensfindung (Methodological Standard for the Development of Quality Indicators within Clinical Practice Guidelines – Results of a structured consensus process).
      ], definitions and terminologies [
      • Bishop D.V.M.
      • Snowling M.J.
      • Thompson P.A.
      • Greenhalgh T.
      Phase 2 of CATALISE: a multinational and multidisciplinary Delphi consensus study of problems with language development: Terminology.
      ], white papers [
      • Centeno C.
      • Sitte T.
      • de Lima L.
      • Alsirafy S.
      • Bruera E.
      • Callaway M.
      • Foley K.
      • Luyirika E.
      • Mosoiu D.
      • Pettus K.
      • Puchalski C.
      • Rajagopal M.R.
      • Yong J.
      • Garralda E.
      • Rhee J.Y.
      • Comoretto N.
      White Paper for global palliative care advocacy: recommendations from a PAL-LIFE expert advisory group of the Pontifical Academy for Life, Vatican City.
      ], reporting guidelines [
      • Swart E.
      • Bitzer E.M.
      • Gothe H.
      • Harling M.
      • Hoffmann F.
      • Horenkamp-Sonntag D.
      • Maier B.
      • March S.
      • Petzold T.
      • Röhrig R.
      • Rommel A.
      • Schink T.
      • Wagner C.
      • Wobbe S.
      • Schmitt J.
      A consensus German reporting standard for secondary data analyses, version 2 (STROSA-STandardisierte BerichtsROutine für SekundärdatenAnalysen).
      ], treatment methods or health-related interventions [
      • Harwood R.
      • Allin B.
      • Jones C.E.
      • Whittaker E.
      • Ramnarayan P.
      • Ramanan A.V.
      • Kaleem M.
      • Tulloh R.
      • Peters M.J.
      • Almond S.
      • Davis P.J.
      • Levin M.
      • Tometzki A.
      • Faust S.N.
      • Knight M.
      • Kenny S.
      • Agbeko R.
      • Aragon O.
      • Baird J.
      • Bamford A.
      • Bereford M.
      • Bharucha T.
      • Brogan P.
      • Butler K.
      • Carroll E.
      • Cathie K.
      • Chikermane A.
      • Christie S.
      • Clark M.
      • Deri A.
      • Doherty C.
      • Drysdale S.
      • Duong P.
      • Durairaj S.
      • Emonts M.
      • Evans J.
      • Fraser J.
      • Hackett S.
      • Hague R.
      • Heath P.
      • Herberg J.
      • Ilina M.
      • Jay N.
      • Kelly D.
      • Kerrison C.
      • Kraft J.
      • Leahy A.
      • Linney M.
      • Lyall H.
      • McCann L.
      • McMaster P.
      • Miller O.
      • O’Riordan S.
      • Owens S.
      • Pain C.
      • Patel S.
      • Pathan N.
      • Pauling J.
      • Porter D.
      • Prendergast A.
      • Ravi K.
      • Riorden A.
      • Roderick M.
      • Scholefield B.R.
      • Semple M.G.
      • Sen E.
      • Shackley F.
      • Sinha I.
      • Tibby S.
      • Verganano S.
      • Welch S.B.
      • Wilkinson N.
      • Wood M.
      • Yardley I.
      A national consensus management pathway for paediatric inflammatory multisystem syndrome temporally associated with COVID-19 (PIMS-TS): results of a national Delphi process.
      ], recommendations for action by policy-makers or other societal actors [
      • Syed A.M.
      • Camp R.
      • Mischorr-Boch C.
      • Houÿez F.
      • Aro A.R.
      Policy recommendations for rare disease centres of expertise.
      ], and quality indicators for health-related interventions [
      • Boulkedid R.
      • Abdoul H.
      • Loustau M.
      • Sibony O.
      • Alberti C.
      Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review.
      ]. The findings of Delphi studies often have far-reaching consequences for health and patient care. In particular, the epistemic claim of a purely knowledge- or experience-based discourse determines the probability of a relatively large acceptance of the results in healthcare practice. A transparent and critical presentation of the Delphi process and the generation of results can contribute considerably to quality assurance and increasing the acceptance of Delphi techniques.

      Reporting quality of Delphi techniques

      The implementation and data quality of Delphi techniques are rarely discussed or evaluated epistemologically, methodologically or empirically [
      • Diamond I.R.
      • Grant R.C.
      • Feldman B.M.
      • Pencharz P.B.
      • Ling S.C.
      • Moore A.M.
      • Wales P.W.
      Defining consensus: A systematic review recommends methodologic criteria for reporting of Delphi studies.
      ,
      • Boulkedid R.
      • Abdoul H.
      • Loustau M.
      • Sibony O.
      • Alberti C.
      Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review.
      ,
      • Jünger S.
      • Payne S.A.
      • Brine J.
      • Radbruch L.
      • Brearley S.G.
      Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review.
      ,
      • Landeta J.
      Current validity of the Delphi method in social sciences.
      ,
      • Guzys D.
      • Dickson-Swift V.
      • Kenny A.
      • Threlkeld G.
      Gadamerian philosophical hermeneutics as a useful methodological framework for the Delphi technique.
      ]. To date, there are criteria for assessing the quality of Delphi processes in general [
      • Nasa P.
      • Jain R.
      • Juneja D.
      Delphi methodology in healthcare research: How to decide its appropriateness.
      ], but which aspects of a Delphi technique should be reported in scientific publications still remains largely unclear. Niederberger and Spranger [
      • Niederberger M.
      • Spranger J.
      Delphi technique in health sciences: A Map, Front.
      ] describe the common publication practice of Delphi studies in the health sciences in an overview of twelve systematic reviews (with a total of 883 publications). This overview shows that most publications report on central aspects of Delphi techniques, namely the selection of experts, the definition of consensus, the number of rounds and the response rates. However, it also reveals a certain heterogeneity in reporting quality. For example, a clear statement concerning the applied Delphi variant is found in the systematic reviews in 61% to 88% of the Delphi articles examined. Also varied was the availability of information about participants’ expertise or the selection strategy. In a systematic review from the palliative field, almost one in three articles showed deficiencies in the presentation of questionnaire development [
      • Jünger S.
      • Payne S.A.
      • Brine J.
      • Radbruch L.
      • Brearley S.G.
      Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review.
      ]. In another review from the health science sector, the number was much higher at 93% [
      • Boulkedid R.
      • Abdoul H.
      • Loustau M.
      • Sibony O.
      • Alberti C.
      Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review.
      ]. Modification or adaptation of items during the Delphi process is reported by between 59% [
      • Diamond I.R.
      • Grant R.C.
      • Feldman B.M.
      • Pencharz P.B.
      • Ling S.C.
      • Moore A.M.
      • Wales P.W.
      Defining consensus: A systematic review recommends methodologic criteria for reporting of Delphi studies.
      ] and 100% [
      • Sinha I.P.
      • Smyth R.L.
      • Williamson P.R.
      Using the Delphi technique to determine which outcomes to measure in clinical trials: Recommendations for the future based on a systematic review of existing studies.
      ] of the publications reviewed. References to feedback design are found in 38% [
      • Humphrey-Murto S.
      • Varpio L.
      • Wood T.J.
      • Gonsalves C.
      • Ufholz L.-A.
      • Mascioli K.
      • Wang C.
      • Foth T.
      The use of the Delphi and other consensus group methods in medical education research: a review.
      ] to 55% [
      • Foth T.
      • Efstathiou N.
      • Vanderspank-Wright B.
      • Ufholz L.-A.
      • Dütthorn N.
      • Zimansky M.
      • Humphrey-Murto S.
      The use of Delphi and Nominal Group Technique in nursing education: A review.
      ] of the articles in the systematic reviews. Possible reasons for these differences in reporting quality have not yet been systematically investigated. They are probably due to field-specific practices, publisher requirements, and the lack of a comprehensively and generally accepted reporting guidelines for studies using Delphi techniques.

      Reporting guidelines for Delphi procedures in the health sciences

      Reporting guidelines are becoming more and more established in the health sciences in order to standardize and improve the reporting of empirical studies. They typically include a list of criteria that should be addressed in a report based on a particular research method. In particular, the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) network has set out to improve the quality of scientific publications in the health sciences by providing a collection of reporting guidelines. Reporting guidelines now exist for various types of studies and their application has become established internationally (e.g., CONSORT, CONsolidated Standards of Reporting Trials [
      • Schulz K.F.
      • Altman D.G.
      • Moher D.
      CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials.
      ]). There is also evidence that reporting guidelines considerably improve the reporting quality of empirical studies [
      • Plint A.C.
      • Moher D.
      • Morrison A.
      • Schulz K.
      • Altman D.G.
      • Hill C.
      • Gaboury I.
      Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review.
      ].
      The objective of the Delphi technique (such as consensus building [
      • Bishop D.V.M.
      • Snowling M.J.
      • Thompson P.A.
      • Greenhalgh T.
      Phase 2 of CATALISE: a multinational and multidisciplinary Delphi consensus study of problems with language development: Terminology.
      ,
      • Swart E.
      • Bitzer E.M.
      • Gothe H.
      • Harling M.
      • Hoffmann F.
      • Horenkamp-Sonntag D.
      • Maier B.
      • March S.
      • Petzold T.
      • Röhrig R.
      • Rommel A.
      • Schink T.
      • Wagner C.
      • Wobbe S.
      • Schmitt J.
      A consensus German reporting standard for secondary data analyses, version 2 (STROSA-STandardisierte BerichtsROutine für SekundärdatenAnalysen).
      ] or foresight [
      • Franklin K.K.
      • Hart J.K.
      Idea generation and exploration: benefits and limitations of the Policy Delphi research method.
      ]) varies between (and also within) subfields of the health sciences and determines the design of the Delphi process, e.g. in terms of group size and level of heterogeneity or the type of knowledge (level of expertise) and how it is asked for [
      • Beiderbeck D.
      • Frevel N.
      • von der Gracht H.A.
      • Schmidt S.L.
      • Schweitzer V.M.
      Preparing, conducting, and analyzing Delphi surveys: Cross-disciplinary practices, new directions, and advancements.
      ]. There are various proposals for reporting guidelines for Delphi studies in the health science sector; one has already been published in the EQUATOR network [
      • Jünger S.
      • Payne S.A.
      • Brine J.
      • Radbruch L.
      • Brearley S.G.
      Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review.
      ]. However, it remains unclear if these guidelines are applicable across subfield boundaries, for example, when we think of more natural science fields such as nutrition science and more social science fields such as the sociology of health. One previously published reporting guideline is based only on studies in palliative care and their primary objective of consensus building [
      • Jünger S.
      • Payne S.A.
      • Brine J.
      • Radbruch L.
      • Brearley S.G.
      Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review.
      ]. Subfield-specific differences can also be observed in other parameters of the Delphi process, for example, determining the level of consensus. For developing medical guidelines, the level of consensus is defined [

      German Association of the Scientific Medical Societies, AWMF Guidance Manual and Rules for Guideline Development, 2013.

      ], whereas for other objectives and fields no recommendations of consensus indicators exist [
      • Niederberger M.
      • Köberich S.
      members of the DeWiss Network, Coming to consensus: The Delphi technique.
      ]. In the health sciences, consensus is defined primarily in terms of percentage agreement but also by central tendency, formal measure of agreement (e.g. Crohnbach’s alpha, Kendal’s W) or a combination of several measures [
      • Nasa P.
      • Jain R.
      • Juneja D.
      Delphi methodology in healthcare research: How to decide its appropriateness.
      ,
      • Diamond I.R.
      • Grant R.C.
      • Feldman B.M.
      • Pencharz P.B.
      • Ling S.C.
      • Moore A.M.
      • Wales P.W.
      Defining consensus: A systematic review recommends methodologic criteria for reporting of Delphi studies.
      ].
      In the health sciences in general there is a broad variation regard to the use and valuation of methods (qualitative or quantitative) and in the analysis of data (such as statistical tests in relation to the scale length, procedure for analyzing responses of open-ended question) [
      • Ioannidis J.P.A.
      Meta-research: Why research on research matters.
      ]. These different research approaches also differ in their epistemological grounding, as they are anchored in different theories of science and knowledge cultures (e.g., natural science versus social science) [
      • Reichmann W.
      Epistemic participation: How to produce knowledge about the economic future.
      ,
      • Jünger S.
      Delphi-Verfahren in den Gesundheitswissenschaften – erkenntnistheoretische Potenziale und Herausforderungen.
      ,
      • Blumer H.
      Symbolic interactionism: perspective and method.
      ,
      • Popper K.R.
      The logic of scientific discovery.
      ]. This also holds true for the Delphi technique. In general, the Delphi technique is rooted in different epistemological traditions and these epistemological groundings also vary with the specific type of Delphi that is applied [
      • Guzys D.
      • Dickson-Swift V.
      • Kenny A.
      • Threlkeld G.
      Gadamerian philosophical hermeneutics as a useful methodological framework for the Delphi technique.
      ,
      • Jünger S.
      Delphi-Verfahren in den Gesundheitswissenschaften – erkenntnistheoretische Potenziale und Herausforderungen.
      ,
      • Scheele D.
      Reality Construction as a Product of Delphi Interaction.
      ]. Thus, a reporting guideline must take this diversity into account. The aim of this study is to identify the similarities, differences, and possible shortcomings of existing reporting guidelines, particularly regarding epistemological and methodological aspects, and to draft an initial proposal for a comprehensively applicable reporting guideline in the health science sector.

      Methodical approach

      We conducted a methodological review to analyze existing reporting guidelines on Delphi techniques in the health sciences. As no reporting guideline has yet been published on this, we have used the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analysis) statement as guidance for reporting [
      • Page M.J.
      • Moher D.
      • Bossuyt P.M.
      • Boutron I.
      • Hoffmann T.C.
      • Mulrow C.D.
      • Shamseer L.
      • Tetzlaff J.M.
      • Akl E.A.
      • Brennan S.E.
      • Chou R.
      • Glanville J.
      • Grimshaw J.M.
      • Hróbjartsson A.
      • Lalu M.M.
      • Li T.
      • Loder E.W.
      • Mayo-Wilson E.
      • McDonald S.
      • McGuinness L.A.
      • Stewart L.A.
      • Thomas J.
      • Tricco A.C.
      • Welch V.A.
      • Whiting P.
      • McKenzie J.E.
      PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews.
      ]. The review was not pre-registered and no study protocol is available. For our methodological review it was not possible to apply all of the topics and proposed analyses included in the PRISMA statement (prior definition of data items, study risk of bias assessment, effect measures, quantitative synthesis methods und results (e.g., sensitivity or subgroup analyses, summary measures, meta-regression), certainty assessment, registration and protocol) mainly due to the examined datatypes, the inductive approach, and the accuracy of fit of the guideline to the to the methodological focus of the study.

      Literature search - Identification of the reporting guidelines

      The identification of existing reporting guidelines on Delphi techniques in the health sciences was done in five different ways. Table 1 gives an overview.
      • 1.
        To create an overview [
        • Niederberger M.
        • Spranger J.
        Delphi technique in health sciences: A Map, Front.
        ] of reporting practices for Delphi studies, two of the authors (MN, JS) identified systematic reviews of Delphi studies in the health sciences in 2019. For the current study, the reviews included in the overview (n = 12) were searched for reporting guidelines by two of the authors (MN, JS) independently using full text screening. Reviews without recommendations on reporting were excluded.
      • 2.
        One author (JS) performed a supplemental search. The initial search yielded 139 hits, which were screened by one author (JS) based on titles and abstracts for a methodological focus of the studies and recommendations on reporting Delphi studies. Primary studies that seemed to report only results of their own Delphi process were excluded. The full texts of those considered potentially relevant were screened by one author (JS) for reporting guidelines on Delphi studies.
      • 3.
        On April 21, 2021, the scientific Delphi expert network (https://delphi.ph-sg.de/) conducted a literature search on Delphi studies in central data resources for the social and health sciences (Scopus, MEDLINE, CINAHL, and Epistemonikos). The German-speaking scientific network DeWiss consists of 20 researchers from different subfields and epistemological backgrounds and focuses on epistemological foundations, methodological issues and guidelines for the implementation and reporting of Delphi studies. The aim of the literature search was to create two data resources on 1) Delphi primary studies and 2) methodological studies on Delphi techniques (available via https://www.zotero.org/groups/4396781/dewiss_datenbanken_delphi-studien). The inclusion criteria for building the data resources are briefly described below. Original papers with Delphi studies were assigned to the two data resources via title-abstract screening by six researchers (MN, SD, JS) and research fellows (MT, LS, XS) of the Delphi network without verification by a second reviewer. A total of 155 articles were assigned to the database with methodological studies (2) and 7,044 articles to the data resource of Delphi primary studies (1). In the current study, a sub-analysis was performed in the two data resources. The data resource of methodological studies (2) was searched via title-abstract-screening for recommendations for reporting Delphi studies by a research fellow (MT) and the full texts of those considered potentially relevant were screened by one of the authors (JS). The filtered data resource of Delphi primary studies (1) (see Table 1) yielded in n = 1,683 articles. The title, abstract and the full texts of those considered potentially relevant were screened for reporting recommendations by one of the authors (JS) and a research fellow (XS), not independently. Studies that seemed to report only the results of their own Delphi process were excluded.
      • 4.
        In addition, the authors performed a search of the EQUATOR network data resource. One author (JS) screened the hits for recommendations on reporting Delphi studies.
      • 5.
        We also included articles in the screening process, which had come to attention by members of the Delphi expert network (DeWiss) in the course of their own research or through scientific networks. All of the authors screened the full texts of the proposed articles for recommendations on reporting Delphi studies.
      Table 1Overview of data resources and search strategies.
      No.Data resourcesSearch strategy
      1Overview (previous research) conducted in PubMed in April 2019
      • Niederberger M.
      • Spranger J.
      Delphi technique in health sciences: A Map, Front.
      Keywords: “review” and “delphi”, English or German language, no time limit.
      2Supplementary search of the overview (see No. 1) in PubMed conducted in June 2021Keywords: (Delphi[Title]) AND ((Guideline[Title]) OR (Guidelines[Title]) OR (Reporting[Title]))English or German language, no time limit.
      3Sub-analysis of the Delphi expert network (DeWiss) data resources on 1) Delphi primary studies and 2) methodological studiesconducted in Scopus, MEDLINE, CINAHL, Epistemonikosin April 20211) primary studies: keywords “delphi*”, “reporting” and “guideline” in title or abstract, English or German language, from 2016 to April 2021.

      2) methodological studies: keyword “delphi*” in title or abstract, English or German language, from 2016 to April 2021.
      4EQUATOR network search conducted in April 2021Keyword: Delphi
      5Contacting experts in January 2022Manual search of the Delphi expert network (DeWiss) members in the course of their own research

      Content analysis of existing reporting guidelines

      We conducted a content analysis that included all reporting recommendations in the manuscripts [

      P. Mayring, Qualitative content analysis: theoretical foundation, basic procedures and software solution, Klagenfurt, 2014.

      ]. In doing so, we pursued two objectives: 1) to provide an overview of the range of dimensions in the reporting guidelines; 2) to reveal the frequencies of the dimensions mentioned. Here, content analysis is suitable for both qualitative and quantitative evaluation and presentation of results [

      P. Mayring, Qualitative content analysis: theoretical foundation, basic procedures and software solution, Klagenfurt, 2014.

      ]. In a first step, two researchers (JS, AH) independently paraphrased, summarized and structured all relevant text passages close to the original text. They formed paraphrases from the text material and inductively determined categories from them. The independent procedure was intended to achieve as much intersubjectivity as possible [
      • Page M.J.
      • Moher D.
      • Bossuyt P.M.
      • Boutron I.
      • Hoffmann T.C.
      • Mulrow C.D.
      • Shamseer L.
      • Tetzlaff J.M.
      • Akl E.A.
      • Brennan S.E.
      • Chou R.
      • Glanville J.
      • Grimshaw J.M.
      • Hróbjartsson A.
      • Lalu M.M.
      • Li T.
      • Loder E.W.
      • Mayo-Wilson E.
      • McDonald S.
      • McGuinness L.A.
      • Stewart L.A.
      • Thomas J.
      • Tricco A.C.
      • Welch V.A.
      • Whiting P.
      • McKenzie J.E.
      PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews.
      ]. In a second step, a third researcher (MN) compared the category systems of the two researchers and in the case of divergent coding, moderated the exchange between the researchers. The goal was to consensually merge both category systems into a unified system. In a third step, all of the paraphrases were reviewed again and classified in the category system. It was also noted to which publications the respective paraphrases could be assigned. It became apparent that some paraphrases could not be clearly assigned to a specific category. In these cases, the research team (JS, AH, MN) discursively agreed on an assignment. For the presentation of the frequency, we determined the number of publications that contained at least one statement (paraphrase) on the respective subcategory (dimension).

      Proposal for a Delphi reporting guideline

      Based on the synthesis of existing reporting guidelines and against the background of the Delphi expert network’s discussion about the epistemological foundations of the Delphi technique, we drafted an initial proposal for Delphi reporting guidelines that is applicable regardless of variant, objective or subfield and reflected on possible limitations of the analyzed recommendations on reporting Delphi studies. Scientists from the Delphi expert network reviewed and consented to this proposal and reflected on its coverage for subfields in the health sciences.

      Results

      Literature search

      We identified a total of ten articles with reporting guidelines for Delphi studies in the health science sector (see Table 2). Figure 1 gives an overview of the results of the literature search.
      Table 2Overview of reporting guidelines for Delphi procedures.
      IDAuthorsYearJournalTopicMethodologyResearch goal
      1Boulkedid et al.2011PLOS ONEHealth servicesSystematic review including 80 Delphi studiesDescribe how the selection of health quality indicators should be reported in Delphi studies
      2Diamond et al.2014Journal of Clinical EpidemiologyNot limitedSystematic review, random sample of 100 Delphi studiesDescribe how to clearly document consensus in Delphi studies
      3Hasson et al.2000Journal of Advanced NursingNursing professionTheoretical considerations of 3 experts (authors)Provide health professionals with an understanding of the preparation, action steps, and difficulties of Delphi studies
      4Humphrey-Murto et al.2017Academic MedicineMedical education researchTheoretical considerations of authors with examples from the relevant research literature.Suggestions (“how to use guide”) that researchers should consider when planning and publishing studies using consensus group methods
      5Humphrey-Murto et al.2019Journal of RheumatologyClinical trials in rheumatic diseasesEvaluation of a preliminary developed checklist by 170 experts (clinicians, researchers, patients, pharmaceutical representatives, fellows and regulatory authorities)Provide a guide on how to clearly outline the multiple aspects of the Delphi process
      6Iglesias et al.2016Pharmaco EconomicsModel-Based economic evaluations of healthcare interventionsTwo-round online Delphi process with 15 experts with previous experience on studies to identify expert judgments in the context of economic evaluations in healthcareProduce guidelines for reporting expert elicitation studies and Delphi studies to capture expert opinion
      7Jünger et al.2017Journal of Palliative MedicinePalliative careSystematic review including 30 Delphi studiesDevelop best practice guidelines for reporting Delphi studies
      8Sinah et al.2011PLOS MedicineClinical trialsSystematic review including 15 Delphi studiesDevelop a core set of outcomes for presenting measurement in clinical trials
      9Toma & Picioreanu2016International Journal of Public Health ResearchPrimary careTheoretical considerations of authors with examples from the relevant research literature.Describe the methodological steps, identify systematic bias, and propose a reporting and reading template for Delphi studies.
      10Waggoner et al.2016Academic MedicineNot limitedLiterature review including 50 studies using consensus methodologyGuidelines for describing the use of a consensus methodology (Delphi, nominal group process or consensus development panel)
      Figure thumbnail gr1
      Figure 1Overview of the results of the systematic literature search.
      Four of the twelve reviews included in the overview [
      • Niederberger M.
      • Spranger J.
      Delphi technique in health sciences: A Map, Front.
      ] contained specific recommendations for reporting Delphi results. The supplemental search enabled the inclusion of one additional study. Four articles could be included in the search within the two data resources of the Delphi expert network. No further hits were detected by searching the EQUATOR network. Table 2 gives an overview of all publications that contain explicitly formulated guidelines for reporting. Studies that report only the results of their own Delphi process and not recommendations on reporting Delphi studies were excluded (e.g. [
      • Sforzini L.
      • Worrell C.
      • Kose M.
      • Anderson I.M.
      • Aouizerate B.
      • Arolt V.
      • Bauer M.
      • Baune B.T.
      • Blier P.
      • Cleare A.J.
      • Cowen P.J.
      • Dinan T.G.
      • Fagiolini A.
      • Ferrier I.N.
      • Hegerl U.
      • Krystal A.D.
      • Leboyer M.
      • McAllister-Williams R.H.
      • McIntyre R.S.
      • Meyer-Lindenberg A.
      • Miller A.H.
      • Nemeroff C.B.
      • Normann C.
      • Nutt D.
      • Pallanti S.
      • Pani L.
      • Penninx B.W.J.H.
      • Schatzberg A.F.
      • Shelton R.C.
      • Yatham L.N.
      • Young A.H.
      • Zahn R.
      • Aislaitner G.
      • Butlen-Ducuing F.
      • Fletcher C.
      • Haberkamp M.
      • Laughren T.
      • Mäntylä F.-L.
      • Schruers K.
      • Thomson A.
      • Arteaga-Henríquez G.
      • Benedetti F.
      • Cash-Gibson L.
      • Chae W.R.
      • de Smedt H.
      • Gold S.M.
      • Hoogendijk W.J.G.
      • Mondragón V.J.
      • Maron E.
      • Martynowicz J.
      • Melloni E.
      • Otte C.
      • Perez-Fuentes G.
      • Poletti S.
      • Schmidt M.E.
      • van de Ketterij E.
      • Woo K.
      • Flossbach Y.
      • Ramos-Quiroga J.A.
      • Savitz A.J.
      • Pariante C.M.
      A Delphi-method-based consensus guideline for definition of treatment-resistant depression for clinical trials.
      ]).

      Content analysis

      We inductively formed nine main categories: Justification, Expert panel, Questionnaire, Survey design, Process regulation, Analyses, Results, Discussion, Methods reflection & Ethics. Each of the main categories had two to five subcategories. The paraphrasing shows that the range of aspects within the subcategories differs considerably.
      The frequencies show that some subcategories are addressed only by single reporting guidelines (ID number indicates order in which the publication is listed in Table 2): meeting of participants (ID1), proceeding with the survey method (ID7), transfer of the results (ID7), validation (ID7), prevention of bias (ID7). Only the subcategory “consensus” is mentioned in all ten publications examined. Most of the reporting guidelines refer to the study objective (n = 6 (ID1, ID3, ID4, ID6, ID7, ID9)), the expert panel (expert selection n = 9 (all except ID1) and panel size and composition n = 9 (all except ID2)), the development of the questionnaire (n = 7 (ID1, ID4-ID9)), the number and design of rounds (n = 7 (ID1, ID3-ID7, ID10)), the feedback design (n = 7 (ID1, ID4-ID9)), the response rate (n = 6 (ID3, ID4, ID6-ID9)), and the form in which the results are presented (n = 7 (ID1, ID3, ID4, ID6-ID9)). Reflections on the epistemological foundations of the Delphi technique, however, are only mentioned in one publication (ID7) and even there with only a minor reference to the constructivist character of Delphi techniques (see Table 3).
      Table 3Category system.
      Main categorySubcategoryFrequency
      Refers to the number of listed publications (Table 2) in which this subcategory was mentioned;
      Paraphrases
      I Justification

      The reporting guidelines require disclosure of the research question or the objective of the Delphi techniques, the question of how the evidence base was taken into account, and the justification for choosing the Delphi technique.
      Rationale5
      • -
        Rationale for the choice of Delphi (ID3,ID4,ID5,ID6,ID7)
        ID number indicates order in which the publication is listed in Table 2.
      • -
        Topic justification (ID3,ID6)
      • -
        Keeping Delphi’s constructivist nature in mind (ID7)
      Study objective6
      • -
        Aim/purpose of the study (ID1,ID4,ID7,ID9)
      • -
        Research problem clearly defined (ID3,ID6)
      • -
        Explicit research question (ID6)
      • -
        Suitability of study objective and Delphi (ID4,ID6,ID7)
      Explanation of the Delphi method5
      • -
        Methodological objective (ID2)
      • -
        Clear explanation of the method (ID3,ID6)
      • -
        Justification for any modification to the Delphi technique (ID4,ID7)
      Evidence on the topic4
      • -
        Literature review (ID3,ID4,ID6)
      • -
        Topic under study (ID3)
      • -
        Evidence base for the topic (ID6,ID7)
      II Expert panel

      It is recommended to report on the selection of the experts, the composition, the sample size, the approach, the handling of anonymity and necessary background information.
      Expert selection9
      • -
        Selection criteria (ID2,ID4,ID7,ID6)
      • -
        Selection/sampling process (ID3,ID8,ID9)
      • -
        Rationale for inclusion/exclusion (ID2,ID4,ID5,ID9)
      • -
        Definition of expert (ID4,ID6,ID9,ID10)
      Expert recruitment5
      • -
        Geographic scope (ID1)
      • -
        Methods of recruitment (ID1,ID6,ID7,ID8,ID10)
      Anonymity4
      • -
        Level of anonymity (ID8)
      • -
        Maintaining anonymity (ID4,ID5)
      • -
        Acknowledgement of anonymity (ID6)
      Panel size and composition9
      • -
        Expert characteristics (ID1,ID3,ID4)
      • -
        Panel composition (ID1,ID6,ID8,ID10)
      • -
        Number of participants, panel size (ID5,ID8,ID10)
      • -
        Method for determining the sample size (ID5)
      • -
        Number of groups in each step (ID4,ID5)
      • -
        Sociodemographic details (ID7)
      • -
        Information on expertise (ID7,ID9)
      • -
        Types of participants (ID8)
      • -
        Willingness to participate (ID9)
      Provided materials5
      • -
        Background information provided (if necessary, per group of participants) (ID4,ID5,ID9)
      • -
        Information/material provided before the first round (ID7,ID8)
      III Questionnaire

      It is recommended to describe the survey instrument; this concerns the structure, the scope, and the research questions on which basis the items were formulated.
      Questionnaire development7
      • -
        Item generation for the first questionnaire (ID1,ID4,ID5,ID8)
      • -
        Way of formulating questions (ID1)
      • -
        Selection of items, criteria (ID1,ID4,ID5)
      • -
        Types of questions used (ID6)
      • -
        Design of the survey instruments (ID7, ID9)
      • -
        Scale/response options (number, order, labeling) (ID1,ID6,ID9)
      Questionnaire data4
      • -
        Copy of the questionnaires (ID1,ID3,ID6)
      • -
        Number of items in the first questionnaire (ID1,ID9)
      • -
        Number of issues generated (ID3)
      • -
        What was asked in each round? (ID8)
      IV Survey design

      Indication of the number of rounds and presentation of the feedback design are recommended, partly also the duration of the Delphi procedure and the termination criterion.
      Rounds7
      • -
        Number of rounds (ID1,ID3,ID4,ID6,ID7)
      • -
        Outline of each round (ID3)
      • -
        A priori decision of round number? (ID5)
      • -
        Plans for moving from one round to the next (ID6)
      • -
        Explanation of reason for more than two rounds (ID10)
      • -
        Design of rounds (ID7)
      Meeting of participants1
      • -
        Was a meeting held? (ID1)
      • -
        Description of meeting participants (ID1)
      Feedback design7
      • -
        Mode of feedback to experts after each round (ID1,ID4,ID5,ID6,ID8)
      • -
        Quantitative and qualitative feedback (each round) and overall group response (ID9)
      • -
        Is feedback given to the group and/or individual? (ID6)
      • -
        Processing and synthesis of experts’ responses to inform the subsequent round (ID7)
      Stopping criterion5
      • -
        Stopping criteria (in absence of consensus) (ID2,ID4,ID9)
      • -
        Threshold to terminate the process (ID2,ID7)
      • -
        Terminating criteria when not determined a priori (ID5)
      Duration2
      • -
        Duration of the Delphi procedure (ID1,ID9)
      V Process regulation

      In part, criteria are formulated for the survey mode and the development of the items. Single reporting guidelines list the expert panel's reflections on the various Delphi rounds, especially from the perspective of dissent, as well as information on a pretest.
      Pretest2
      • -
        Piloting of material and survey instruments (ID4,ID7)
      • -
        Reviewing and piloting background information in advance (ID7)
      Data collection5
      • -
        Questionnaire administration (e.g., postal) (ID1,ID8,ID9)
      • -
        Software for administration (ID5)
      • -
        Data collection (ID6)
      Proceeding with items5
      • -
        Flow of items eliminated and added in each round (ID1)
      • -
        Criteria for dropping items (ID2)
      • -
        Criteria to include terms/items in subsequent rounds (ID5,ID9)
      • -
        Proceeding with items or topics in the next round (ID7)
      Proceeding with non-responders3
      • -
        Invitation/exclusion of non-responders in subsequent rounds (ID5,ID8,ID9)
      • -
        Management of non-responders (ID5)
      Proceeding with the survey method1
      • -
        Modification of the survey instrument and methodological decisions by the research team based on previous rounds (ID7)
      VI Analyses

      All reporting guidelines call for the definition and handling of consensus.
      Methods of data analysis5
      • -
        Method justification (ID3)
      • -
        Management of (minority) opinions (ID6,ID9)
      • -
        Analyzing and handling qualitative and quantitative data (ID6,ID9)
      • -
        Preparation of a pre-specified data analysis plan (ID6)
      • -
        Methods of data analysis (ID7)
      • -
        Statistical analysis for each question (ID10)
      Definition and dealing with consensus10
      • -
        Definition of consensus (ID1,ID2,ID4,ID5,ID7,ID8,ID9)
      • -
        Issues of consensus (ID3)
      • -
        Interpretation of consensus (ID3)
      • -
        Description of the meaning of consensus in relation to the study aim (ID6)
      • -
        Definition of agreement (ID6)
      • -
        A priori definition (ID7,ID8)
      • -
        Procedure when consensus is reached (ID7)
      • -
        Explanation of what constituted consensus (ID10)
      • -
        Explanation of how consensus was assessed (ID10)
      Dealing with dissent3
      • -
        Interpretation of consensus not reached (ID3)
      • -
        Are outliers to the overall consensus asked for the reasons for their answers? (ID6)
      • -
        Procedures when consensus is not reached (ID7)
      • -
        Strategies to deal with non-consensus (ID7)
      VII Results

      The reporting guidelines require the disclosure of response rates and the presentation of results differentiated according to the various Delphi rounds. The presentation of the methodological procedure in graphic form, e.g., a flowchart, is also recommended.
      Response rate6
      • -
        Response rates in each round (if necessary, per group of participants) (ID3,ID4,ID6,ID7,ID8,ID9)
      • -
        Non-response rate (ID7)
      • -
        Number who completed every round (ID8)
      Reporting results7
      • -
        List of selected items (ID1)
      • -
        List of all outcome participants agreed/selected (ID1,ID8,ID9)
      • -
        Results of each score (ID1)
      • -
        Results for each round (ID3,ID4,ID7,ID8)
      • -
        Presentation of results in round 2, indicating the strength of support in round (ID3)
      • -
        Summary of the total number of issues generated in round 1 (ID6)
      • -
        Summary of the strength of overall consensus in the final round (ID6)
      • -
        Graphic and/or statistical representations (ID6)
      • -
        Figures showing the average group response (ID7)
      • -
        Changes between rounds (ID7)
      • -
        Distribution of each outcome in the final round (ID8)
      Reporting the overall process4
      • -
        Diagram of participant flow (ID1)
      • -
        Outline of the overall process, description of each step (ID5)
      • -
        Flow chart for illustration, including preparatory phase, rounds, participants, interim and concluding steps (ID4,ID7)
      VIII Discussion

      Data are interpreted and other uses discussed; dissemination of the results and further research needs should be identified.
      Interpretation of results3
      • -
        Statistical interpretation (ID3)
      • -
        Interpretation guideline for the reader (ID3,ID6)
      • -
        Presentation of the meaning of the results (ID6)
      • -
        How to digest the findings in relation to the emphasis being placed upon them (ID6)
      • -
        Interpretation of consensus (not necessarily imply the correct answer) and non-consensus (provide informative insights and highlight differences in perspectives) (ID7)
      Transfer of the results1
      • -
        Clearly identifiable resulting guidance (ID7)
      • -
        Recommendations for transfer into practice and implementation (ID7)
      • -
        Reference to a more detailed presentation (ID7)
      Dissemination plan2
      • -
        Dissemination plan to facilitate implementation (ID7)
      • -
        Were the final results of the study sent to the panel experts? (ID9)
      Further research2
      • -
        Direction of future research (ID3,ID6)
      IX Methods reflection & Ethics

      The results are to be discussed with regard to the quality of the data and possible limitations or biases. In some reporting guidelines it is recommended to disclose in the report how the anonymity of the respondents was handled and whether ethical aspects were taken into account in the survey process.
      Validation1
      • -
        Review of the final draft by an external board or authority before publication (ID7)
      Validity/Reliability2
      • -
        Identification of reliability and validity issues (ID3,ID6)
      Discussion of limitations4
      • -
        Discussion of methodological issues (ID4,ID5)
      • -
        Recognition of the influence of the composition of the panel on the results (ID6)
      • -
        Limitation of reporting data from quantitative questions (ID6)
      • -
        Critical reflection on potential limitations and their impact (ID4,ID7)
      • -
        Discussion of lack of consensus to inform the future research agenda (ID4)
      • -
        Adequacy of conclusions (ID7)
      Prevention of bias1
      • -
        Examining the effect of background information on expert judgment (ID7)
      • -
        Measures to avoid directly or indirectly influencing experts’ judgments (ID7)
      Further ethical issues2
      • -
        Ethical responsibilities/issues towards experts, sample and research community (ID3,ID6)
      1 Refers to the number of listed publications (Table 2) in which this subcategory was mentioned;
      2 ID number indicates order in which the publication is listed in Table 2.

      Proposal for a Delphi reporting guideline

      The challenge in preparing a comprehensive Delphi reporting guideline is to consider the diversity of Delphi variants and the respective modifications of the Delphi technique. Involving other scientists from the Delphi expert network, we formulated an initial proposal for Delphi reporting guidelines in Figure 2 based on an epistemological discussion and synthesis of existing reporting guidelines. These criteria appear central to reporting on Delphi studies regardless of variant, objective or subfield.
      Figure thumbnail gr2
      Figure 2Proposal for reporting guidelines on Delphi techniques in the health sciences.

      Discussion

      The reporting guidelines analyzed here relate to Delphi studies in health sciences. None of the identified reporting guidelines claim to cover the entire Delphi process or the spectrum of Delphi variants or their possible applications in the different health science subfields. As a consequence, the criteria are divergent and only the aspect of consensus appears in all of the reporting guidelines. In some cases, the information in the reporting guidelines is imprecisely formulated because it is not specified whether the information refers to the beginning of the Delphi technique, the process or the results. Especially information about the expert panel or the questionnaire can change during the Delphi study. Sometimes it remains unclear when methodological decisions have to be made or defined. This particularly concerns the consensus definition, which can be made a priori or a posteriori.
      The reporting guidelines examined were developed on the basis of Delphi studies that followed more of a traditional approach, such as maintaining anonymity or multiple completed rounds. Hence, the variety of Delphi variants described in the introductory section is not sufficiently considered. This is critical because some Delphi variants break up core characteristics of Delphi techniques, such as anonymity in a group Delphi or the round-based principle in the real-time Delphi. This means that in some cases other aspects have to be reported for different Delphi variants [
      • Hasson F.
      • Keeney S.
      • McKenna H.
      Research guidelines for the Delphi survey technique.
      ,
      • Jünger S.
      • Payne S.A.
      • Brine J.
      • Radbruch L.
      • Brearley S.G.
      Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review.
      ]. However, even in classical Delphi studies, e.g. in the medical field [
      • Nasa P.
      • Jain R.
      • Juneja D.
      Delphi methodology in healthcare research: How to decide its appropriateness.
      ], basic methodological characteristics of the Delphi process, such as iterative rounds and the receiving of feedback, are not always given in studies labeled as Delphi technique. In relation to the research goal, Delphi studies are conducted in healthcare particularly to identify consensus [
      • Niederberger M.
      • Spranger J.
      Delphi technique in health sciences: A Map, Front.
      ,
      • Jorm A.F.
      Using the Delphi expert consensus method in mental health research.
      ,
      • Nasa P.
      • Jain R.
      • Juneja D.
      Delphi methodology in healthcare research: How to decide its appropriateness.
      ], but as described in the introductory section, Delphi techniques are applicable to a variety of objectives [
      • Niederberger M.
      • Spranger J.
      Delphi technique in health sciences: A Map, Front.
      ,
      • Beiderbeck D.
      • Frevel N.
      • von der Gracht H.A.
      • Schmidt S.L.
      • Schweitzer V.M.
      Preparing, conducting, and analyzing Delphi surveys: Cross-disciplinary practices, new directions, and advancements.
      ]. A comprehensive reporting guideline could make Delphi studies with different subfield-specific objectives comparable, including in terms of their quality. Likewise, it can highlight the variety of possible applications and complexity of the process. A Delphi process includes several steps of preparing, conducting and analyzing and is thus also more complex and time-consuming than single-stage survey procedures. Thus, a minimum of 16 weeks is recommended for the Delphi process alone, with an additional six to ten weeks for writing a scientific manuscript [
      • Beiderbeck D.
      • Frevel N.
      • von der Gracht H.A.
      • Schmidt S.L.
      • Schweitzer V.M.
      Preparing, conducting, and analyzing Delphi surveys: Cross-disciplinary practices, new directions, and advancements.
      ].
      Also, the discussion about the epistemological foundations of the Delphi technique in general and of the specific, applied Delphi variant in particular is often neglected and is not adequately considered in the existing reporting guidelines (see Table 3). One reason could be that the practical implications cannot be derived directly. However, the question of whether the Delphi technique is based on a realist perspective following critical rationalism or on a constructivist perspective is relevant for the evaluation of the quality of a Delphi process and its results. We assume that critical rationalism is more prevalent in the natural sciences and constructivism is more prevalent in the social sciences. This means:
      • According to critical rationalism [
        • Popper K.R.
        The logic of scientific discovery.
        ], a Delphi technique should be able to produce an approximation of “true” or objective knowledge by correcting the possibly biased expert judgments through feedback from the respondent group. Thus, a consensus among the participants can be interpreted as an approximation of “true” knowledge.
      • From a constructivist perspective [
        • Blumer H.
        Symbolic interactionism: perspective and method.
        ], there is no “true” knowledge, only “situated” knowledge which is produced, applied and further developed in social interactions.
      For example, Campbell [
      • Campbell S.M.
      How do stakeholder groups vary in a Delphi technique about primary mental health care and what factors influence their ratings?.
      ] showed in a methodological study on the Delphi technique that different stakeholder panels in primary mental health care “have diverse views of quality of care and these differences translate into how they rate quality indicators” [
      • Campbell S.M.
      How do stakeholder groups vary in a Delphi technique about primary mental health care and what factors influence their ratings?.
      ]. From a constructivist perspective, this comes not as a surprise and would also not be regarded as a problem. Form a critical rationalist perspective, however, this can be as rendered as failure with regard to establishing consensus and approximating a “true” value. Thus, depending on the epistemological perspective, one can arrive at opposite conclusions concerning the quality of the respective Delphi results.
      The bottom line is that reporting guidelines encourage researchers to reflect on and disclose key assumptions and methodological choices, as well as the epistemological foundations on which basis results are interpreted. The criteria of the newly developed proposal for Delphi reporting guidelines (see Fig. 2) are not to be seen as completely independent of each other. From a methodological perspective they are closely interrelated and must form a coherent frame. This concerns, in particular, the fit of the epistemological positioning, the weighting of qualitative or quantitative aspects, and the quality criteria. Concerning all criteria, it is also important to disclose the time of definition or determination during the Delphi techniques. Furthermore, all criteria should be reflected throughout the individual Delphi rounds.

      Limitations

      There are also at least three potential limitations concerning this study, which should be mentioned here. The screening and inclusion of articles was partly based on the decision of a single researcher. For resource reasons, an independent assessment of multiple researchers, as it was applied for the content analysis, was not possible at all steps. The inclusion of further recommendations from the Delphi expert network should take this limitation into account and draw attention to possibly missed key articles in the research field. Furthermore, this proposal for a Delphi reporting guideline cannot be assumed in terms of international representativeness. Neither the analyzed guidelines nor the Delphi expert network could fulfill these requirements. However, the group of authors as well as the Delphi expert network consists of experts from different subfields and ranges of experience in the application of the Delphi technique (e.g., concerning different Delphi variants). On this point, the heterogeneity of expertise was considered. Finally, the reporting and conducting of the study could have been improved with an appropriate reporting guideline for methodological reviews and an a priori published study protocol.

      Conclusion

      The methodological review provides an overview of existing reporting guidelines for Delphi studies in the health sciences. The content analysis shows that they vary widely, both in terms of criteria and level of detail. Most reporting guidelines explicitly address reporting on consensus definition, expert panel, response, feedback design, and number of rounds. However, they lack differentiation, for example, with regard to different Delphi variants and objectives. In particular, epistemological principles have been neglected so far, even though they are central to assessing the quality of the procedure and classifying the collected data. Based on a synthesis of existing reporting guidelines and in discussion with experts from different subfields with particular consideration of epistemological questions, we have developed an initial proposal for a comprehensive reporting guideline for the health sciences. The results of our methodological review do not provide a final reporting guideline, but it is an important step toward improving the quality of Delphi techniques. The different criteria need to be further differentiated (in an expert discourse).

      Funding

      The DeWiss Network is supported by the German research foundation (Project Number: 429572724). The study is part of the project DELPHISTAR – Delphi studies in social and health sciences – recommendations for a standardized reporting. The project has been registered at the Equator network platform and the OSF platform and can be assessed at this page: https://osf.io/gc4jk.

      Acknowledgements

      For their support in the screening process we would like to thank Stefanie Deckert, Leonie Salzmann, Xenia Sestak and Michaela Trompke.

      Conflict of interest

      The authors declare that there are no conflicts of interest.

      CRediT author statement

      Marlen Niederberger: Conceptualization, analysis of the reporting guidelines, writing (original draft) and corrections.
      Julia Spranger: Research and analysis of the reporting guidelines, writing (original draft) and corrections.
      Angelika Homberg: Analysis of the reporting guidelines, writing (original draft) and corrections.
      Marco Sonnberger: Writing (original draft) and corrections.
      Members of DeWiss: Consulting.

      References

        • Linstone H.A.
        • Turoff M.
        The delphi method.
        Addison-Wesley, MA1975
        • Humphrey-Murto S.
        • Varpio L.
        • Wood T.J.
        • Gonsalves C.
        • Ufholz L.-A.
        • Mascioli K.
        • Wang C.
        • Foth T.
        The use of the Delphi and other consensus group methods in medical education research: a review.
        Acad. Med. 2017; 92: 1491-1498https://doi.org/10.1097/ACM.0000000000001812
        • Hutchings A.
        • Raine R.
        A systematic review of factors affecting the judgments produced by formal consensus development methods in health care.
        J. Health Serv. Res. Policy. 2006; 11: 172-179https://doi.org/10.1258/135581906777641659
        • Niederberger M.
        • Köberich S.
        members of the DeWiss Network, Coming to consensus: The Delphi technique.
        EJCN. 2021; 20: 692-695https://doi.org/10.1093/eurjcn/zvab059
        • Niederberger M.
        • Spranger J.
        Delphi technique in health sciences: A Map, Front.
        Public Health. 2020; 8: 1-10https://doi.org/10.3389/fpubh.2020.00457
        • Jorm A.F.
        Using the Delphi expert consensus method in mental health research.
        Aust. N. Z. J. Psychiatry. 2015; 49: 887-897https://doi.org/10.1177/0004867415600891
        • Nasa P.
        • Jain R.
        • Juneja D.
        Delphi methodology in healthcare research: How to decide its appropriateness.
        WJM. 2021; 11: 116-129https://doi.org/10.5662/wjm.v11.i4.116
        • Diamond I.R.
        • Grant R.C.
        • Feldman B.M.
        • Pencharz P.B.
        • Ling S.C.
        • Moore A.M.
        • Wales P.W.
        Defining consensus: A systematic review recommends methodologic criteria for reporting of Delphi studies.
        J. Clin. Epidemiol. 2014; 67: 401-409https://doi.org/10.1016/j.jclinepi.2013.12.002
        • Hasson F.
        • Keeney S.
        • McKenna H.
        Research guidelines for the Delphi survey technique.
        J. Adv. Nurs. 2000; 32: 1008-1015
        • Gnatzy T.
        • Warth J.
        • von der Gracht H.
        • Darkow I.-L.
        Validating an innovative real-time Delphi approach – A methodological comparison between real-time and conventional Delphi studies.
        Technol. Forecast. Soc. Change. 2011; 78: 1681-1694https://doi.org/10.1016/j.techfore.2011.04.006
        • Franklin K.K.
        • Hart J.K.
        Idea generation and exploration: benefits and limitations of the Policy Delphi research method.
        Innov. High Educ. 2006; 31: 237-246https://doi.org/10.1007/s10755-006-9022-8
        • Seker S.E.
        Computerized Argument Delphi technique.
        IEEE Access. 2015; 3: 368-380https://doi.org/10.1109/ACCESS.2015.2424703
        • Habibi A.
        • Jahantigh F.F.
        • Sarafrazi A.
        Fuzzy Delphi technique for forecasting and screening items, Asia.
        J. Res. Bus. Econ. Manage. 2015; 5: 130-143https://doi.org/10.5958/2249-7307.2015.00036.5
        • Deckert S.
        • Arnold K.
        • Becker M.
        • Geraedts M.
        • Brombach M.
        • Breuing J.
        • Bolster M.
        • Assion C.
        • Birkner N.
        • Buchholz E.
        • Carl E.-G.
        • Diel F.
        • Döbler K.
        • Follmann M.
        • Harfst T.
        • Klinkhammer-Schalke M.
        • Kopp I.
        • Lebert B.
        • Lühmann D.
        • Meiling C.
        • Niehues T.
        • Petzold T.
        • Schorr S.
        • Tholen R.
        • Wesselmann S.
        • Voigt K.
        • Willms G.
        • Neugebauer E.
        • Pieper D.
        • Nothacker M.
        • Schmitt J.
        Methodischer Standard für die Entwicklung von Qualitätsindikatoren im Rahmen von S3-Leitlinien – Ergebnisse einer strukturierten Konsensfindung (Methodological Standard for the Development of Quality Indicators within Clinical Practice Guidelines – Results of a structured consensus process).
        Z. Evid. Fortbild. Qual. Gesundhwes. 2021; 160: 21-33https://doi.org/10.1016/j.zefq.2020.11.008
        • Bishop D.V.M.
        • Snowling M.J.
        • Thompson P.A.
        • Greenhalgh T.
        Phase 2 of CATALISE: a multinational and multidisciplinary Delphi consensus study of problems with language development: Terminology.
        J. Child Psychol. Psychiatry. 2017; 58: 1068-1080https://doi.org/10.1111/jcpp.12721
        • Centeno C.
        • Sitte T.
        • de Lima L.
        • Alsirafy S.
        • Bruera E.
        • Callaway M.
        • Foley K.
        • Luyirika E.
        • Mosoiu D.
        • Pettus K.
        • Puchalski C.
        • Rajagopal M.R.
        • Yong J.
        • Garralda E.
        • Rhee J.Y.
        • Comoretto N.
        White Paper for global palliative care advocacy: recommendations from a PAL-LIFE expert advisory group of the Pontifical Academy for Life, Vatican City.
        J. Palliat. Med. 2018; 21: 1389-1397https://doi.org/10.1089/jpm.2018.0248
        • Swart E.
        • Bitzer E.M.
        • Gothe H.
        • Harling M.
        • Hoffmann F.
        • Horenkamp-Sonntag D.
        • Maier B.
        • March S.
        • Petzold T.
        • Röhrig R.
        • Rommel A.
        • Schink T.
        • Wagner C.
        • Wobbe S.
        • Schmitt J.
        A consensus German reporting standard for secondary data analyses, version 2 (STROSA-STandardisierte BerichtsROutine für SekundärdatenAnalysen).
        Gesundheitswesen. 2016; 78: e145-e160https://doi.org/10.1055/s-0042-108647
        • Harwood R.
        • Allin B.
        • Jones C.E.
        • Whittaker E.
        • Ramnarayan P.
        • Ramanan A.V.
        • Kaleem M.
        • Tulloh R.
        • Peters M.J.
        • Almond S.
        • Davis P.J.
        • Levin M.
        • Tometzki A.
        • Faust S.N.
        • Knight M.
        • Kenny S.
        • Agbeko R.
        • Aragon O.
        • Baird J.
        • Bamford A.
        • Bereford M.
        • Bharucha T.
        • Brogan P.
        • Butler K.
        • Carroll E.
        • Cathie K.
        • Chikermane A.
        • Christie S.
        • Clark M.
        • Deri A.
        • Doherty C.
        • Drysdale S.
        • Duong P.
        • Durairaj S.
        • Emonts M.
        • Evans J.
        • Fraser J.
        • Hackett S.
        • Hague R.
        • Heath P.
        • Herberg J.
        • Ilina M.
        • Jay N.
        • Kelly D.
        • Kerrison C.
        • Kraft J.
        • Leahy A.
        • Linney M.
        • Lyall H.
        • McCann L.
        • McMaster P.
        • Miller O.
        • O’Riordan S.
        • Owens S.
        • Pain C.
        • Patel S.
        • Pathan N.
        • Pauling J.
        • Porter D.
        • Prendergast A.
        • Ravi K.
        • Riorden A.
        • Roderick M.
        • Scholefield B.R.
        • Semple M.G.
        • Sen E.
        • Shackley F.
        • Sinha I.
        • Tibby S.
        • Verganano S.
        • Welch S.B.
        • Wilkinson N.
        • Wood M.
        • Yardley I.
        A national consensus management pathway for paediatric inflammatory multisystem syndrome temporally associated with COVID-19 (PIMS-TS): results of a national Delphi process.
        Lancet Child Adolesc. Health. 2021; 5: 133-141https://doi.org/10.1016/S2352-4642(20)30304-7
        • Syed A.M.
        • Camp R.
        • Mischorr-Boch C.
        • Houÿez F.
        • Aro A.R.
        Policy recommendations for rare disease centres of expertise.
        Eval. Program Plann. 2015; 52: 78-84https://doi.org/10.1016/j.evalprogplan.2015.03.006
        • Boulkedid R.
        • Abdoul H.
        • Loustau M.
        • Sibony O.
        • Alberti C.
        Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review.
        PLoS ONE. 2011; 6: 1-9https://doi.org/10.1371/journal.pone.0020476
        • Jünger S.
        • Payne S.A.
        • Brine J.
        • Radbruch L.
        • Brearley S.G.
        Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review.
        Palliat. Med. 2017; 31: 684-706https://doi.org/10.1177/0269216317690685
        • Landeta J.
        Current validity of the Delphi method in social sciences.
        Technol. Forecast. Soc. Change. 2006; 73: 467-482https://doi.org/10.1016/j.techfore.2005.09.002
        • Guzys D.
        • Dickson-Swift V.
        • Kenny A.
        • Threlkeld G.
        Gadamerian philosophical hermeneutics as a useful methodological framework for the Delphi technique.
        Int. J. Qual. Stud. Health Well-being. 2015; 10: 1-14https://doi.org/10.3402/qhw.v10.26291
        • Sinha I.P.
        • Smyth R.L.
        • Williamson P.R.
        Using the Delphi technique to determine which outcomes to measure in clinical trials: Recommendations for the future based on a systematic review of existing studies.
        PLoS Med. 2011; 8: 1-5https://doi.org/10.1371/journal.pmed.1000393
        • Foth T.
        • Efstathiou N.
        • Vanderspank-Wright B.
        • Ufholz L.-A.
        • Dütthorn N.
        • Zimansky M.
        • Humphrey-Murto S.
        The use of Delphi and Nominal Group Technique in nursing education: A review.
        IJNS. 2016; 60: 112-120https://doi.org/10.1016/j.ijnurstu.2016.04.015
        • Schulz K.F.
        • Altman D.G.
        • Moher D.
        CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials.
        BMJ. 2010; 340https://doi.org/10.1136/bmj.c332
        • Plint A.C.
        • Moher D.
        • Morrison A.
        • Schulz K.
        • Altman D.G.
        • Hill C.
        • Gaboury I.
        Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review.
        Med. J. Aust. 2006; 185: 263-267https://doi.org/10.5694/j.1326-5377.2006.tb00557.x
        • Beiderbeck D.
        • Frevel N.
        • von der Gracht H.A.
        • Schmidt S.L.
        • Schweitzer V.M.
        Preparing, conducting, and analyzing Delphi surveys: Cross-disciplinary practices, new directions, and advancements.
        MethodsX. 2021; 8https://doi.org/10.1016/j.mex.2021.101401
      1. German Association of the Scientific Medical Societies, AWMF Guidance Manual and Rules for Guideline Development, 2013.

        • Ioannidis J.P.A.
        Meta-research: Why research on research matters.
        PLoS Biol. 2018; 16https://doi.org/10.1371/journal.pbio.2005468
        • Reichmann W.
        Epistemic participation: How to produce knowledge about the economic future.
        Soc. Stud. Sci. 2013; 43: 852-877https://doi.org/10.1177/0306312713498641
        • Jünger S.
        Delphi-Verfahren in den Gesundheitswissenschaften – erkenntnistheoretische Potenziale und Herausforderungen.
        in: Niederberger M. Renn O. Delphi-Verfahren in den Sozial- und Gesundheitswissenschaften: Konzept. Varianten und Anwendungsbeispiele, Springer VS, Wiesbaden2019: 55-81
        • Blumer H.
        Symbolic interactionism: perspective and method.
        Englewood Cliffs, New Jersey1969
        • Popper K.R.
        The logic of scientific discovery.
        Routledge, Abingdon, UK2005
        • Scheele D.
        Reality Construction as a Product of Delphi Interaction.
        in: Turoff M. Linstone H.A. The Delphi Method: Techniques and Applications. Addison-Wesley, Boston2002: 35-67
        • Page M.J.
        • Moher D.
        • Bossuyt P.M.
        • Boutron I.
        • Hoffmann T.C.
        • Mulrow C.D.
        • Shamseer L.
        • Tetzlaff J.M.
        • Akl E.A.
        • Brennan S.E.
        • Chou R.
        • Glanville J.
        • Grimshaw J.M.
        • Hróbjartsson A.
        • Lalu M.M.
        • Li T.
        • Loder E.W.
        • Mayo-Wilson E.
        • McDonald S.
        • McGuinness L.A.
        • Stewart L.A.
        • Thomas J.
        • Tricco A.C.
        • Welch V.A.
        • Whiting P.
        • McKenzie J.E.
        PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews.
        BMJ. 2021; 372https://doi.org/10.1136/bmj.n160
      2. P. Mayring, Qualitative content analysis: theoretical foundation, basic procedures and software solution, Klagenfurt, 2014.

        • Sforzini L.
        • Worrell C.
        • Kose M.
        • Anderson I.M.
        • Aouizerate B.
        • Arolt V.
        • Bauer M.
        • Baune B.T.
        • Blier P.
        • Cleare A.J.
        • Cowen P.J.
        • Dinan T.G.
        • Fagiolini A.
        • Ferrier I.N.
        • Hegerl U.
        • Krystal A.D.
        • Leboyer M.
        • McAllister-Williams R.H.
        • McIntyre R.S.
        • Meyer-Lindenberg A.
        • Miller A.H.
        • Nemeroff C.B.
        • Normann C.
        • Nutt D.
        • Pallanti S.
        • Pani L.
        • Penninx B.W.J.H.
        • Schatzberg A.F.
        • Shelton R.C.
        • Yatham L.N.
        • Young A.H.
        • Zahn R.
        • Aislaitner G.
        • Butlen-Ducuing F.
        • Fletcher C.
        • Haberkamp M.
        • Laughren T.
        • Mäntylä F.-L.
        • Schruers K.
        • Thomson A.
        • Arteaga-Henríquez G.
        • Benedetti F.
        • Cash-Gibson L.
        • Chae W.R.
        • de Smedt H.
        • Gold S.M.
        • Hoogendijk W.J.G.
        • Mondragón V.J.
        • Maron E.
        • Martynowicz J.
        • Melloni E.
        • Otte C.
        • Perez-Fuentes G.
        • Poletti S.
        • Schmidt M.E.
        • van de Ketterij E.
        • Woo K.
        • Flossbach Y.
        • Ramos-Quiroga J.A.
        • Savitz A.J.
        • Pariante C.M.
        A Delphi-method-based consensus guideline for definition of treatment-resistant depression for clinical trials.
        Mol. Psychiatry. 2021; https://doi.org/10.1038/s41380-021-01381-x
        • Campbell S.M.
        How do stakeholder groups vary in a Delphi technique about primary mental health care and what factors influence their ratings?.
        Qual. Saf. Health Care. 2004; 13: 428-434https://doi.org/10.1136/qshc.2003.007815