Advertisement

Implementation of patient-reported outcome assessment in routine cancer care: A systematic review of multicentric programs in Europe

  • Author Footnotes
    1 Shared first authorship.
    Madlen Scheibe
    Footnotes
    1 Shared first authorship.
    Affiliations
    Center for Evidence-Based Healthcare, University Hospital Carl Gustav Carus and Carl Gustav Carus Faculty of Medicine, Technische Universität Dresden, Dresden, Germany
    Search for articles by this author
  • Author Footnotes
    1 Shared first authorship.
    Alina Herrmann
    Footnotes
    1 Shared first authorship.
    Affiliations
    Center for Evidence-Based Healthcare, University Hospital Carl Gustav Carus and Carl Gustav Carus Faculty of Medicine, Technische Universität Dresden, Dresden, Germany
    Search for articles by this author
  • Jochen Schmitt
    Affiliations
    Center for Evidence-Based Healthcare, University Hospital Carl Gustav Carus and Carl Gustav Carus Faculty of Medicine, Technische Universität Dresden, Dresden, Germany
    Search for articles by this author
  • Natascha Einhart
    Affiliations
    Center for Evidence-Based Healthcare, University Hospital Carl Gustav Carus and Carl Gustav Carus Faculty of Medicine, Technische Universität Dresden, Dresden, Germany
    Search for articles by this author
  • Brita Sedlmayr
    Affiliations
    Center for Evidence-Based Healthcare, University Hospital Carl Gustav Carus and Carl Gustav Carus Faculty of Medicine, Technische Universität Dresden, Dresden, Germany
    Search for articles by this author
  • Christoph Kowalski
    Correspondence
    Corresponding author. Christoph Kowalski, PhD, German Cancer Society, Kuno-Fischer-Strasse 8, 14057 Berlin, Germany.
    Affiliations
    German Cancer Society, Berlin, Germany
    Search for articles by this author
  • Author Footnotes
    1 Shared first authorship.
Published:September 24, 2020DOI:https://doi.org/10.1016/j.zefq.2020.08.001

      Abstract

      Introduction

      To give an overview of patient-reported outcome measures (PROMs) programs in routine cancer care that allow for both major purposes of PROM assessment: 1) monitoring of an individual patient's outcome to assist treatment decision making, and 2) use in quality improvement initiatives including the benchmarking of providers. We synthesize information on program elements like the mode of assessment and questionnaire used, as well as information relevant for adaptation following a PDCA scheme.

      Method

      We carried out a systematic literature research in the databases PubMed and EMBASE using MeSH terms and keywords related to PROM assessment in routine cancer care to identify eligible studies published between January 2003 and November 2018 (PROSPERO reg. no. CRD42019141402). We included studies in which PROM assessment programs had been reported as being implemented in clinical practice as well as collected multicentrically with at least one site in Europe and in which PROMs had been collected before and at least once after intervention. Study authors were queried to verify or correct the program elements extracted and merged during the review. Study quality assessment was not done, since it is not expedient for the objective of this review.

      Results

      Overall, 5,545 unique references were identified, 5,416 of which were excluded after the screening of titles and abstracts. Of the 29 references assessed, five programs were identified and included in the synthesis. The programs included those from Germany, Austria, Denmark, the Netherlands and the UK, and patients with different cancer types and tumor stages, which used paper-based or purely electronic PROM assessment.

      Discussion and conclusion

      Few reports have so far been published on PROM programs that allow for both the monitoring of an individual patient's outcome and use in quality improvement initiatives. The studies revealed relevant information on existing PROM programs and gave valuable insight into issues that need to be considered when setting up such an infrastructure. Some critical issues, however, were hardly addressed, among them costs, staff resources and methods of reporting and responding.

      Zusammenfassung

      Hintergrund

      Ziel der Arbeit ist es, einen Überblick über Programme zur Erfassung von Patient-Reported Outcome Measures (PROMs) in der klinischen Routinebehandlung von Krebspatientinnen und -patienten zu geben, die beide Hauptzwecke von PROMs ermöglichen: 1) das Monitoring von Symptomen und Funktion einzelner Betroffener zur Unterstützung der Behandlungsentscheidung und 2) die Verwendung der Ergebnisse zur Qualitätsverbesserung, einschließlich des Benchmarkings von Versorgenden. Synthetisiert werden Informationen zu Programmelementen wie Bewertungsmodus und verwendetem Fragebogen sowie Informationen, die gemäß PDCA-Schema relevant sind.

      Methode

      Durchgeführt wurde eine systematische Literaturrecherche in den Datenbanken PubMed und EMBASE unter Verwendung von MeSH-Begriffen und Schlüsselwörtern zur Identifikation von Studien, die zwischen Januar 2003 und November 2018 veröffentlicht wurden (PROSPERO Registrierungsnr. CRD42019141402). Eingeschlossen wurden Studien, in denen über PROM-Programme berichtet wurde, die in der klinischen Praxis implementiert waren, multizentrisch inkl. mindestens einem Standort in Europa, und in denen PROMs vor und mindestens einmal nach einer Intervention gesammelt wurden. Die Autoren der Studie wurden gebeten, die Programmelemente, die im Rahmen des Reviews extrahiert und zusammengeführt wurden, zu überprüfen oder zu korrigieren. Eine Bewertung der Studienqualität wurde nicht durchgeführt, da dies für das Ziel dieser Übersichtsarbeit nicht zweckmäßig ist.

      Ergebnisse

      Initial wurden 5.545 Treffer identifiziert, von denen 5.416 nach Überprüfung der Titel und Abstracts ausgeschlossen wurden. In den 29 untersuchten Volltexten wurden fünf Programme identifiziert und in die Synthese einbezogen. Die Programme stammten aus Deutschland, Österreich, Dänemark, den Niederlanden und dem Vereinigten Königreich und berücksichtigten Patientinnen und Patienten mit unterschiedlichen Krebsarten und Tumorstadien. Verwendet wurden sowohl papierbasierte als auch rein elektronische PROM-Instrumente.

      Diskussion und Schlussfolgerung

      Es konnten nur wenige veröffentlichte Berichte zu PROM-Programmen identifiziert werden, in denen sowohl Symptome und Funktion einzelner Betroffener verfolgt als auch Maßnahmen zur Qualitätsverbesserung ergriffen werden können. Die identifizierten Studien lieferten relevante Informationen zu bestehenden PROM-Programmen und gaben Einblicke in Fragen, die beim Aufbau einer solchen Infrastruktur berücksichtigt werden sollten. Eine Reihe kritischer Themen wurde jedoch kaum angesprochen, darunter Kosten, Personalressourcen und Methoden zur Meldung und Beantwortung.

      Keywords

      Schlüsselwörter

      Introduction

      The benefits of patient-reported outcome measure (PROM) assessment are well described in the literature. To distinguish PROMs from patient-reported experience measures (PREMs) the Organisation for Economic Co-operation and Development (OECD) provided a brief definition focused on health status and excluding satisfaction with care, quality of interaction with providers and other indicators: “PROMs: Measure patients’ perceptions of their health status, clinical outcomes, mobility and quality of life” [

      OECD. RECOMMENDATIONS TO OECD MINISTERS OF HEALTH FROM THE HIGH LEVEL REFLECTION GROUP ON THE FUTURE OF HEALTH STATISTICS. Strengthening the international comparison of health system performance through patient-reported indicators; 2017.

      : p. 25]. Not only do PROMs help healthcare providers to assess patients’ symptoms and functions more precisely [
      • Efficace F.
      • Rosti G.
      • Aaronson N.
      • et al.
      Patient- versus physician-reporting of symptoms and health status in chronic myeloid leukemia.
      ,
      • Fromme E.K.
      • Eilers K.M.
      • Mori M.
      • et al.
      How accurate is clinician reporting of chemotherapy adverse effects?.
      ] and assist with clinical decision making [
      • Basch E.
      • Abernethy A.P.
      Supporting Clinical Practice Decisions With Real-Time Patient-Reported Outcomes.
      ,
      • Klinkhammer-Schalke M.
      • Koller M.
      • Steinger B.
      • et al.
      Direct improvement of quality of life using a tailored quality of life diagnosis and therapy pathway: randomised trial in 200 women with breast cancer.
      ], recent studies also showed improved survival when cancer patients are monitored with PROMs [
      • Basch E.
      • Deal A.M.
      • Dueck A.C.
      • et al.
      Overall Survival Results of a Trial Assessing Patient-Reported Outcomes for Symptom Monitoring During Routine Cancer Treatment.
      ,
      • Denis F.
      • Lethrosne C.
      • Pourel N.
      • et al.
      Randomized Trial Comparing a Web-Mediated Follow-up With Routine Surveillance in Lung Cancer Patients.
      ]. Although evidence for the benefits of PROM from studies is substantial, the issue of how to best implement PROM assessment into routine care is widely unsolved, not only because funding for such programs outside of time-bound research studies is generally difficult to collect, but also because it is not clear what works best in different clinical situations when implementing routine PROM programs. Decisions that need to be made when planning PROM programs include the choice of instruments, the mode of administration (e.g., on-site vs at home, online v paper-based), and feedback provision, among others [
      • Snyder C.F.
      • Aaronson N.K.
      • Choucair A.K.
      • et al.
      Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations.
      ].
      Recent initiatives defined standards for measuring process and outcome quality, including PROMs and guidance on when or how to collect data and with which specific questionnaire, such as the ICHOM Standard Set for Colorectal Cancer [
      • Zerillo J.A.
      • Schouwenburg M.G.
      • van Bommel A.C.M.
      • et al.
      An International Collaborative Standardizing a Comprehensive Patient-Centered Outcomes Measurement Set for Colorectal Cancer.
      ]. Its implementation is still limited to a few sites, typically with variation in data collection that does not allow for the comparison of site outcomes. Large existing programs that collect PROM data include those issued by the National Health Service (NHS) from 2009 [
      • Devlin N.J.
      • Appleby J.
      Getting the most out of PROMs Putting health outcomes at the heart of NHS decision-making. In..
      ], that of the Swedish Hip Arthroplasty Register that has collected PROMs since 2002 [
      • Rolfson O.
      • Karrholm J.
      • Dahlberg L.E.
      • et al.
      Patient-reported outcomes in the Swedish Hip Arthroplasty Register: results of a nationwide prospective observational study.
      ], or initiatives led by charities such as the TrueNTH Global Registry that is funded by the Movember Foundation and that collects outcomes data including PROMs in prostate cancer patients in 15 countries [
      • Evans S.M.
      • Millar J.L.
      • Moore C.M.
      • et al.
      Cohort profile: the TrueNTH Global Registry - an international registry to monitor and improve localised prostate cancer health outcomes.
      ]. Such programs have the explicit aim to compare/benchmark outcomes of providers, to encourage mutual learning, and, in the case of the NHS, to even make parts of reimbursement dependent on PROM results.
      The European Organization for Research and Treatment of Cancer (EORTC) Quality of Life Group, one of the leading PROM developers for cancer, issued a manual for the utilization of PROMs in daily practice [
      • Wintner L.M.
      • Sztankay M.
      • Aaronson N.
      • et al.
      The use of EORTC measures in daily clinical practice—A synopsis of a newly developed manual.
      ]. Similarly, the International Society for Quality of Life Research (ISOQOL) and others discussed logistics and provided recommendations several years earlier [
      • Snyder C.F.
      • Aaronson N.K.
      • Choucair A.K.
      • et al.
      Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations.
      ,
      • Rose M.
      • Bezjak A.
      Logistics of collecting patient-reported outcomes (PROs) in clinical practice: an overview and practical examples.
      ,

      International Society for Quality of Life Research (prepared by Aaronson, N., E.T., Greenhalgh, J., Halyard, M., Hess, R., Miller, D., Reeve, B., Santana, M., Snyder, C)., User's Guide to Implementing Patient-Reported Outcomes Assessment in Clinical, Practice., In; 2015.

      ]; in 2017, Franklin and colleagues [
      • Franklin P.
      • Chenok K.
      • Lavalee D.
      • et al.
      Framework To Guide The Collection And Use Of Patient-Reported Outcome Measures In The Learning Healthcare System.
      ] did likewise, including best practice examples. Similarly, Nordan et al. recently described challenges and success factors for the implementation of electronic PROMs within a single large medical center [
      • Nordan L.
      • Blanchfield L.
      • Niazi S.
      • et al.
      Implementing electronic patient-reported outcomes measurements: challenges and success factors.
      ]. Recommendations cover major aspects such as measure selection, choice of patients addressed, assessment timing as well as scoring and reporting techniques. Whether these recommendations are met in practice and whether they are feasible in routine care, however, is less clear. A number of reviews described routine care approaches to PROM assessment in the past [
      • Snyder C.F.
      • Aaronson N.K.
      • Choucair A.K.
      • et al.
      Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations.
      ] and gave insight into facilitating factors and obstacles under different circumstances. Such knowledge is imperative to derive an implementation strategy that suites the situation, e.g. the clinical setting, the country, the stakeholders involved and much more.
      In this report, we attempt to add to the existing body of knowledge regarding the implementation of PROMs into routine cancer care by conducting a systematic review to identify PROM programs that allow for both major purposes of PROM assessment: 1) the identification of an individual patient's symptoms and function, to assist in communicating and clinical decision making as well as monitoring his or her outcome, and 2) the use of group data in quality improvement initiatives, including the benchmarking of providers or sites with respect to outcomes. This research was conducted as part of Work Package 10 of the iPAAC Joint Action – Innovative Partnership for Action against Cancer, which is co-funded by the Health Programme of the European Union that brings together 24 European countries to develop and implement innovative approaches to cancer control. The search strategy was developed by the IPAAC Work Package 10 consortium and thus reflects the demand of the various professional groups, among them many clinicians (doctors AND nurses), patients, and the countries involved. The aim of Work Package 10 is to “develop practical instruments to support Member States in successful governance of cancer care, ensuring standardized, integrated and comprehensive oncological care” [

      iPAAC Consortium. Innovative Partnership for Action against Cancer (iPAAC) - Official iPAAC Joint Action presentation. In; 2018.

      : p. 14], including the collection of PROMs. The results from this report will guide the development of a framework for the implementation of PROMs in routine cancer care. Further information on the joint action's aims are provided at https://www.ipaac.eu/. To be better able to suggest recommendations for the integration of PROMs in “comprehensive oncological care”, we systematically synthesize attributes of the PROM programs identified in the systematic review, like mode of assessment and instrument used, as well as attributes that are associated with acceptance and usability of the programs, thereby contributing to the success of PROM routines.

      Methods

      Literature search

      We conducted a systematic literature search to identify existing PROM programs in routine cancer care that allow for decision-making by individual cancer patients as well as provider comparisons. Programs were thus limited to those that were multicenter as well as those which collected PROMs before and at least once after an intervention, to allow for both major purposes provider comparison and patient monitoring and because pre-intervention assessment is necessary for the risk-adjustment of outcomes. The criterion “multicentricity” includes the utilization of an identical infrastructure, uniform recruitment standards and data that are or at least can be analyzed across sites to allow for comparability. We further restricted our literature research to programs that already published results to avoid the inclusion of conceptual reports or unsuccessful programs. We intended to identify “programs”, not articles, i.e. if two or more original articles on one program were found, we integrated the information into one program-specific synthesis. Defining the PICOS Model for the study characteristics proved only partly practical. Since we were not looking for intervention effects but for programs themselves, only the Population and Intervention dimensions could be defined satisfactorily, with “P” being “patients with any kind of cancer diagnosis treated in Europe” and “I” being “multicentric implementation of Patient Reported Outcome Measures (PROMs) in routine cancer care with at least one pre- and one post-intervention assessment”. Only primary research Studies were considered, but if reviews were identified, the reviewed research would have been assessed for inclusion. This incomplete PICOS definition was included in the study protocol that also contained specific questions relevant for deriving good practices following a plan-do-check-act-scheme (see results section, Table 3). We excluded time-bound research projects that were funded for only a limited amount of time without clear hints that the underlying PROM infrastructure is already or may be implemented into routine care afterwards. Since this report will be used to derive recommendations for PROMs to be implemented in cancer centers in Europe and data protection regulations substantially differ from other parts of the world, programs were restricted to those conducted in Europe, using a geographical understanding of Europe including countries with only parts of their land mass being part of Europe, such as Turkey, Russia, Georgia, and Azerbaijan. Title/abstracts had to be in English and no further language restrictions were defined for full text language. The search was restricted to articles published from 1 January 2003 to 1 November 2018. The resulting search string is documented in appendix A. To supplement our literature database search, we used the backward snowballing technique and checked the reference lists of the studies identified using our search string. Above that, we carried out a cited reference search to identify relevant studies/papers that have cited the already included studies. Experts from the IPAAC consortium were asked for additional programs. Resulting hits were independently screened by two authors (CK and MS, who was replaced by AH while on leave) after the exclusion of duplicates. In the first step, titles and abstracts were screened. In the case of any disagreements, consensus was reached by discussion. Full texts were retrieved and assessed when both reviewers agreed. If only abstracts were provided, we hand searched full-text publications for follow-up. In cases where articles were published in languages that the review authors could not assess, the corresponding authors were consulted to decide whether the inclusion criteria were met or not. Again, at this stage, in the case of disagreements, consensus was reached by discussion.
      The review was registered at PROSPERO, reg.-no. CRD42019141402.

      Data extraction

      Standardized data abstraction forms were used to depict information on study type, setting, goals (see results section, Table 1), and patient characteristics (number, gender, ethnicity, cancer type, stage) (Table 2), as well as qualitative and quantitative information on 20 specific research questions to identify facilitators and obstacles to implementation and acceptance by those involved (Table 3). Items to identify such potential facilitators and obstacles were derived from existing reviews and the above-mentioned manuals/recommendations following a PDCA-scheme. Two authors independently extracted the identified studies for program attributes in duplicate. In the case of disagreements, results were discussed until consensus was reached. To verify the identified program attributes, corresponding authors and/or program directors were queried to check for correctness of the abstracted information and to provide further information in case none was identified in the program report. Study quality was not assessed since it is not expedient for the objective of this review; that is, we were not interested in the quality of the published articles but rather the attributes of the programs.
      Table 1Study characteristics.
      Nguyen-Nielsen et al., 2016Meisner et al., 2011Wintner et al., 2015Zeegers et al., 2010Engelen et al., 2012
      Study nameThe Danish prostate cancer databasebenchmarking: how to measure outcome quality at the comprehensive oncology center in Stuttgartevaluation of electronic patient-reported outcome assessment with cancer patients in the hospital and at homeThe West Midlands Bladder Cancer Prognosis Programme: rationale and designreporting health-related quality of life scores to physicians during routine follow-up visits of pediatric oncology patients: Is it effective?
      CountryDenmarkGermanyAustriaUnited KingdomNetherlands
      Time period of PROM collectionMay 2011 – April 2015 (ongoing)September 2003 - December 2008 (ongoing)February - September 2012 (ongoing)December 2005 – unclear (ongoing)March 2006 - November 2009 (ongoing)
      Primary research purpose“systematically collecting key clinical variables for the purposes of health care monitoring, quality improvement, and research”evaluation and improvement of outcomescollect information on patients’ internet use and their attitudes towards electronic QoL assessmentobjective is to relate health-related quality of life to the recurrence and progression of bladder cancerinvestigate the effectiveness of an intervention that provides HRQOL scores of the patient to the pediatric oncologist
      Study setting (+ number of follow-ups)population based registry; clinical data collected by treating physicians in 22 urological and oncological departments. PROM data collection at diagnosis and at 1 and 3 yearsPROM assessment in 13 comprehensive oncology centers in the region of Stuttgart, Germany, between 30-09-2003 – 31-12-2008; PROM collection annually after diagnosis; first PROM collection N/Aassessment of patients’ QOL before a follow-up / treatment appointment in a hospital setting (clinic-ePRO), and second, as regular monitoring of the symptom burden of chemotherapy outpatients in the home setting (home-ePRO); two Austrian hospitals; follow-up N/AHematuria clinics within the West Midlands, assessment of QLQ-C30 questionnaire at baseline in the entire cohort, first follow-up at 3 months after diagnosis, from then up to 5 years after diagnosis4 University Medical Centers, pediatric oncology patients and their parents surveyed within three months after diagnosis and then all three months; set of surveys tailored to patient group
      Type of studydisease-specific registry supplemented by PROM data“prospective, non-randomized, multi-centered, registry-based cohort study”observational prospective studyepidemiological prospective cohort studysequential cohort design (first control, then intervention)
      Measuring instrumentN/AEORTC Quality of Life QLQ-C30, Version3.0EORTC QLC-C30general cancer questionnaire QLQ-C30 with the addition of disease-specific assessments using the QLQ-C30, the QLQ-BLS24, with 24 questions specific to NMIBC, the EORTC QLQ-BLM30, with 30 questions specific to muscle-invasive bladder cancerPedsQL self-report, PedsQL proxy report, TAPQOL
      Table 2Sample characteristics.
      Patient characteristicsNguyen-Nielsen et al.,

      2016
      Meisner et al., 2011Wintner et al., 2015Zeegers et al., 2010Engelen et al., 2012
      The Danish prostate cancer databaseBenchmarking: how to measure outcome quality at the comprehensive oncology center in StuttgartEvaluation of electronic patient-reported outcome assessment with cancer patients in the hospital and at homeThe West Midlands Bladder Cancer Prognosis Programme: rationale and designReporting health-related quality of life scores to physicians during routine follow-up visits of pediatric oncology patients: Is it effective?
      Number

      of patients
      22,332 patients3,213 patients with breast cancer,

      216 patients with colon cancer,

      847 rectum carcinoma patients
      113 patients on-site,

      45 patients at home
      771 patients193 patients
      Gendermalemale and femalemale and femaleanyany
      EthnicityN/AN/AN/AN/Aany
      Cancer typeincident prostate cancerbreast, colon, rectum cancergastrointestinal tumors, glioma, gynecological tumors, lung cancer, neuroendocrine tumors, testicular cancerpathologically confirmed urothelial carcinoma of the bladderall cancers, children 0-18 yrs
      StageanyUICC TNM stages 0-IIIN/AanyN/A
      Table 3Qualitative synthesis of program characteristics.
      Research questionsNguyen-Nielsen et al., 2016Meisner et al., 2011Wintner et al., 2015Zeegers et al., 2010Engelen et al., 2012
      The Danish prostate cancer databaseBenchmarking: how to measure outcome quality at the comprehensive oncology center in StuttgartEvaluation of electronic patient-reported outcome assessment with cancer patients in the hospital and at homeThe West Midlands Bladder Cancer Prognosis Programme: rationale and designReporting health-related quality of life scores to physicians during routine follow-up visits of pediatric oncology patients: Is it effective?
      Quotes taken from the author query response. Full documentation of query response for Engelen et al. in appendix A and the answers of Wintner et al. (2015) in appendix A.
      PLAN - Preparation:
      1 Who were the primary drivers of program implementation? (for example, doctors, nurses, patients, quality initiatives, other)N/AN/A“Home ePRO resulted from a longstanding initiative of a chief physician at the Kufstein County Hospital and clinic-ePRO at the Medical University of Innsbruck was part of a larger research project initiated by doctors and scientists”
      Quotes taken from the author query response. Full documentation of query response for Engelen et al. in appendix A and the answers of Wintner et al. (2015) in appendix A.
      Cancer Research UK Bladder Cancer Group, University of Birmingham„[D]uring the initial effectiveness and implementation study, researchers were the primary drivers of the program.”
      2 What were the primary purposes of program implementation? (for example, patient monitoring, treatment planning, benchmarking, quality assurance, other)recording of adverse events following treatmentevaluation and improvement of outcome quality; benchmarking outcomes between centersassessment of patients’ Quality of life (QOL)to study the determinants of recurrence and progression of NMIBC and to design a prognostic tool that could predict adverse outcomes; “The research objectives in this second project are to study the effects of recurrence and progression on quality of life; to study the effects of repeat cystoscopy, urostomy and self-catheterization on quality of life; and to study the patients’ assessments of a hypothetical prognostic model and how this affects their preference for the mode of surveillance”„Monitoring quality of life and screening for psychosocial issues in order to assure patient-centered communication during the consultation, identify patients’ needs and refer to necessary intervention if needed.”
      3 How are the patients selected? (i.e., choice of population, sampling strategy, who recruits patients)no selection; All Danish patients with histologically verified prostate cancer are includedpatients treated in participating centerspatients were approached before appointments or treatment application in the case of chemotherapy treatmentpatients: Potentially eligible patients are being identified at hematuria clinics on the basis of abnormal cystoscopic findings suggestive of bladder cancer; population selection: N/A“[A]ll patients with cancer (aged 0-18 years) are approached within one month post-diagnosis and their PROMs are being monitored and discussed every 3 months across the illness trajectory. The implementation process is part of standard care of the hospital and there is a PROM coordinator and support team available that approaches all patients.”
      4 Are proxy respondents allowed?N/AN/Astudy nurses could be asked for helpN/A„Yes; parents (for children aged 0-7 or for mentally disabled children 0-18 years) [..]”
      5 How where the questionnaire(s) used selected? (possible criteria: validity, feasibility, languages available, other)N/AN/A“The EORTC PRO measures are valid, reliable and feasible questionnaires covering a broad variety of aspects relevant for cancer patients and available in many different languages”
      Quotes taken from the author query response. Full documentation of query response for Engelen et al. in appendix A and the answers of Wintner et al. (2015) in appendix A.
      N/A„Mainly standardized questionnaires are selected that are available in Dutch, least burdensome for patients/parents to complete, and applicable to a broad age range. We always start with a standard battery of questionnaires [..]. In addition to this, at the beginning of starting to implement PROMs in a new disease group or population, the PROM coordinator and support team sit together with the team to decide on which questionnaires to use in their clinical practice.“
      6 How extensive are the resources necessary for PROM implementation and maintenance of the program (e.g. staff, costs)? And can recommendations for minimum resources made?N/AN/Astaff available: study nurse; costs: N/AN/A„Currently, 1 PROM coordinator and 3 PROM administrative support persons are employed on the implementation and maintenance of the PROM portal. At a minimum, one PROM coordinator per center is recommended. Furthermore, a flexible technology system needs to be available in order to ensure adaptability of PROMs to contextual needs. Ideally, the PROM system can be linked to/integrated in the electronic health record.“
      7 Is there an estimate on the overall cost per patient for program participation?N/AN/Astaff required: not specified, but study nurse provided an initial training if necessary and further assistance if patients reported any problem with the device or the questionnaire; a study nurse gathered sociodemographic and clinical data from the hospital records and entered them in the database to link with associated QOL data stored in the PROM database; costs: N/Aneed for additional human resources in each participating urology unit; A fully funded study-dedicated research nurse has therefore been provided to each participating unit. The BCPP Research Nurses are highly trained in all aspects of the program and undertake all the extra duties required, without adversely affecting clinicians’ workload and the normal pathways of clinical careN/A
      8 Which conditions had to be fulfilled in advance of the implementation (e.g. necessary IT infrastructure, data security issues, acceptance among healthcare providers and patients, selection and training of staff) - can recommendations be derived?N/AN/AIT infrastructure: if necessary, an Apple iPad 2 was provided for duration of study participation; specialized software for electronic PRO data collection (“CHES”), result calculation, and presentation for use in clinical practice;

      data security: Physicians can use CHES via their work desktop, patients are equipped with login data for access to questionnaires via a web-site or an iOS app
      training of staff: program-specific training of research nurse necessary, but not further specified“All healthcare providers are trained before they start with the use of PROMs in their clinical practice. Please see
      • Santana M.J.
      • Haverman L.
      • Absolom K.
      • et al.
      Training clinicians in how to use patient-reported outcome measures in routine clinical practice.
      for the most up to date description of the training. [..] IT infrastructure: Since July 1st 2019, the PROM system is linked with a viewer-function (F2F) in the electronic health record, such that healthcare providers can immediately look up PROM data without using a password and external website.”

      Data security issues and solutions have been described by
      • Haverman L.
      • van Oers H.A.
      • Limperg P.F.
      • et al.
      Implementation of electronic patient reported outcomes in pediatric daily clinical practice: The KLIK experience.
      .
      DO - Implementation:
      9 What kind of PROMs have been implemented?
      • InstrumentN/AEORTC Quality of Life QLQ-C30, Version 3.0EORTC Quality of Life QLQ-C30EORTC QLQ-C30, QLQ-BLS24, QLQ-BLM30“During treatment:

      About the patient: Questions about school, open question to be asked to healthcare provider, PedsQL generic self-report (8-18 years), PedsQL proxy report (0-7 years), PedsQL Cancer Module self-report (8-18 years), PedsQL Cancer module proxy report (0-7 years)

      About the parent/the family: Psychosocial assessment tool (PAT), Distress Thermometer for Parents (DT-P)



      After end of treatment:

      About the patient: Questions about school, open question to be asked to healthcare provider, PedsQL generic (same as described above), PedsQL self-report fatigue (8-18 years), PedsQL proxy report fatigue (0-7 years)

      About the parent: Distress Thermometer for Parents (DT-P)”
      • generic vs. disease-specificgeneric QOL-measurements: pain level, physical activity, sexual function, depression, urine and fecal incontinencegeneric QOL-measurementsgeneric aspects of QOLgeneric: cancer questionnaire EORTC QLQ-C30



      in addition:

      QLQ-BLS24, with 24 questions specific to NMIBC

      QLQ-BLM30, with 30 questions specific to muscle-invasive bladder cancer
      generic
      • Scaling:

      single vs. multi-item-scale
      no information available on single or multi-item-scalesmulti-item-scalesmulti-item-scaleN/AN/A
      • Response formatN/Apaper-baseddigital via tablet PCN/AN/A
      10 In what phases of the course of diagnoses and treatment have the PROMs been implemented?at diagnosis and at 1 and 3 years follow-upannually after diagnosisbefore each treatment application or appointmentat time of diagnosis, 3 months after diagnosis, then annually up to 5 years“Starting at one month post-diagnosis and continuing every 3 months at outpatient visit until survivorship.”
      11 How were the PROS collected?

      • survey mode

      (face-to-face, online)




      N/A




      postal mailings




      online (self-assessment)




      PROMs collected by trained research nurses via face-to-face interviews




      “Went from digital (QLIC-ON study) to online Portal ([..] www.hetklikt.nu).”
      • frequency and timing of assessmentsN/AN/Abefore each consultation / treatmentat time of diagnosis;

      before transurethral resection of bladder tumor
      “Tied to a medical appointment at the outpatient clinic […] Parents screened for psychosocial issues: once in every 6 months.”
      • intervalsN/Aannuallydepending on consultations/treatment frequencyN/AN/A
      • interview locationN/AN/Ain hospital,

      at home
      N/AN/A
      12 How where the collected data processed (including aids to facilitate score interpretation) and how did they fit into clinical workflow (fed into the hospital management system, shared and subsequently used and by whom, assistance in interpretation)?N/APROM usage for benchmarking purposes; shared and used for discussion in internal workshops for quality improvement/ optimization of clinical careworkflow: Computer-based Health Evaluation System (CHES) as specialized software for electronic PRO data collection, result calculation, and presentation for use in clinical practice; usage by clinicians, who had access to patients’ self-reports via their work desktop, but no advice was given on how to use PRO data for patient appointments; data is stored at each hospital separatelyN/APROM “[…] results discussed once in every 3 months.” (original answer to question 11)

      “PROfile presents 4 HRQOL domains (physical, emotional, social, and cognitive functioning) to pediatrician by summarizing the answers of child or parent, pediatricians are provided with PRO-reports with answers and graphical representations prior to consultations.

      Since 2011, healthcare providers could log on to the website and open the electronic PROfile,

      Since July 2019: healthcare providers can open the electronic PROfile through a viewer-function in the electronic health record.”
      13 Are PROMs used for Benchmarking purposes between care providers and if so, how?N/Ayes, annual cancer-site specific benchmarking feedback to participating sites with n≥100; discussed among colleagues in workshopsprobably not, because data are stored at each hospital separatelyN/A„N/A

      Yet, we currently work with the quality improvement team of the hospital to see if we can include patient-reported experience measures (PREMs), such as the PedsQL healthcare satisfaction hematology-oncology module. This will then be used for internal care experiences and quality improvement.”
      14 What were strategies for responding to issues identified by the questionnaires?N/Adiscussion in workshops“No advice was given on how to use PRO data for patient appointments”
      Quotes taken from the author query response. Full documentation of query response for Engelen et al. in appendix A and the answers of Wintner et al. (2015) in appendix A.
      N/Aa decision tree pocket-card has been developed
      • Santana M.J.
      • Haverman L.
      • Absolom K.
      • et al.
      Training clinicians in how to use patient-reported outcome measures in routine clinical practice.
      ,
      • Engelen V.
      • Haverman L.
      • Koopman H.
      • et al.
      Development and implementation of a patient reported outcome intervention (QLIC-ON PROfile) in clinical paediatric oncology practice.
      .
      STUDY
      Barriers and enablers:
      15 Can recommendations to influence clinical uptake of PROMs in routine care be derived? (e.g. necessary IT infrastructure, data security issues, acceptance among health care providers and patients, sociodemographic of users, costs, time needed, suitable survey instruments)N/AN/Abarriers for patients: internet access, security concerns (older patients) regarding the transmission of health-related data and QOL via the internet; sociodemographic barriers: age; acceptance among health care providers not reportedN/A“Please see our published paper about the implementation process
      • Schepers S.A.
      • Sint Nicolaas S.M.
      • Haverman L.
      • et al.
      Real-world implementation of electronic patient-reported outcomes in outpatient pediatric cancer care.
      and the discussion of the thesis of Schepers (p. 179)
      • Schepers S.A.
      Changing pediatric cancer care: development and implementation of electronic patient and parent reported outcomes.
      .“
      16 What can be recommended for a multicentric implementation of PROMs in comparison to unicentric implementation? What were the lessons learned?N/AN/Anot really multicentric in that no central database was usedN/A„Starting bottom-up (involving the team) works better than top-down; Start small, learn from the implementation process, and slowly start to spread the intervention further; A local PROM coordinator and champion are essential; Always involve the end-users in the process of implementation; Adaption is key (e.g. selection of questionnaires/making it available in electronic health record) [..]”
      Impact and lessons learned:
      17 What was the impact of routine use of PROMS on outcomes at the patient, provider, and system level? How has it been assessed/evaluated? What were the results? Was the aim achieved and did any unintended effects occur?N/AN/AN/A, but a patient evaluation was done to identify attitudes towards ePROMsN/APRO tools facilitate discussion of HRQOL issues and increase physician's awareness of their patients HRQOL; there was also an impact on physician–patient communication with better HRQOL and emotional functioning for some patients; “[W]e showed initial effectiveness as described in [..]
      • Engelen V.
      • Detmar S.
      • Koopman H.
      • et al.
      Reporting health-related quality of life scores to physicians during routine follow-up visits of pediatric oncology patients: Is it effective?.
      . Yet, we also showed that when implementing PROMs in a real-world pediatric oncology setting, challenges arise
      • Schepers S.A.
      • Sint Nicolaas S.M.
      • Haverman L.
      • et al.
      Real-world implementation of electronic patient-reported outcomes in outpatient pediatric cancer care.
      , which might impact the originally intended effect of the intervention (i.e. implementation fidelity). Therefore, continuously monitoring the implementation process, and applying implementation strategies, even in the maintenance phase, is very important.”
      18 What were the lessons learned from PROM implementation?N/AN/A“Involvement of stakeholders and education of patients regarding ePRO and its purpose may ease implementation; common barriers persist (experiencing ePRO as too impersonal or not adequate for the patient's actual situation)”
      Quotes taken from the author query response. Full documentation of query response for Engelen et al. in appendix A and the answers of Wintner et al. (2015) in appendix A.
      N/Amethod was more time-consuming than expected and caused logistical problems
      • Haverman L.
      Electronic patient and parent reported outcomes in pediatric clinical practice. In.
      . “Furthermore, in order for it to be aligned with the clinical workflow, it needs to be integrated in the electronic health record and a PROM coordinator is of utmost importance
      • Schepers S.A.
      • Sint Nicolaas S.M.
      • Haverman L.
      • et al.
      Real-world implementation of electronic patient-reported outcomes in outpatient pediatric cancer care.
      .”
      ACT
      19 Was PROM assessment modified following the results of the initial implementation?N/AN/A“No”
      Quotes taken from the author query response. Full documentation of query response for Engelen et al. in appendix A and the answers of Wintner et al. (2015) in appendix A.
      N/A“[..] The IT system (www.hetklikt.nu) furthermore received an update in 2013, such that we can now flexibly use the portal for the purpose of clinical practice (feedback of PROM results to authorized users), research (no feedback of PROM results), and clinical trials (data extraction for authorized trial data managers). Patients do not need to complete questionnaires twice for different purposes if the timeframes of clinic/research/trials overlap.

      Finally, the pediatric oncology hospital (Princess Máxima Center for pediatric oncology) has established a viewer-function in the electronic health record since July 2019.”
      20 Is there anything else that should be mentioned regarding the PROM program?“The implementation process of the PROM portal is constantly monitored and is dynamic. For pediatric oncology, we moved from an externally research-funded program towards a program that is fully funded by the hospital (since 2018) and is part of standard care.”
      * Quotes taken from the author query response. Full documentation of query response for Engelen et al. in appendix A and the answers of Wintner et al. (2015) in appendix A.

      Results

      The flowchart according to PRISMA [
      • Moher D.
      • Liberati A.
      • Tetzlaff J.
      • et al.
      Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
      ] (Figure 1) depicts the study selection process. In total, 5,545 records were screened, of which 5,516 were excluded during title/abstract screening. Due to the sensitive search string at this stage, records were excluded for a multitude of reasons, most notably: reports on clinical trials investigating the effects of medical treatments, conducted outside, single-center descriptions of survey results Europe (not reported). Among the records excluded at this stage were 32 conference abstracts with no apparent reason for exclusion besides the unavailability of a full text. A hand search was conducted and the authors were contacted via e-mail to identify potentially available full texts. These attempts did not result in records that met the eligibility criteria. The remaining 29 full texts were then assessed for eligibility. Five programs were included in the qualitative synthesis [
      • Engelen V.
      • Detmar S.
      • Koopman H.
      • et al.
      Reporting health-related quality of life scores to physicians during routine follow-up visits of pediatric oncology patients: is it effective?.
      ,
      • Meisner C.
      • Brinkmann F.
      • Henke D.
      • et al.
      [Benchmarking: how to measure outcome quality at the comprehensive oncology centre in Stuttgart].
      ,
      • Nguyen-Nielsen M.
      • Hoyer S.
      • Friis S.
      • et al.
      The Danish Prostate Cancer Database.
      ,
      • Wintner L.M.
      • Giesinger J.M.
      • Zabernigg A.
      • et al.
      Evaluation of electronic patient-reported outcome assessment with cancer patients in the hospital and at home.
      ,
      • Zeegers M.P.
      • Bryan R.T.
      • Langford C.
      • et al.
      The West Midlands Bladder Cancer Prognosis Programme: rationale and design.
      ]. Table 1 presents the study characteristics, Table 2 presents the sample characteristics of the identified articles, and Table 3 presents the findings of the qualitative synthesis and the author query.
      Figure thumbnail gr1
      Figure 1Flowchart of the study selection process.

      Study Characteristics

      The studies were published between 2010 and 2016. Programs were from Denmark, Germany, Austria, the United Kingdom, and the Netherlands; none were cross-national. Studies described interim results for patient samples, which were typically patients recruited in a very early stage after program initiation, and all programs were ongoing at the time of review. Specification of the primary research purpose varied and centered either around quality improvement/evaluation of outcomes, health care monitoring, or the prediction of disease progression/research. The study settings ranged from population-based registries in which all patients were included and into which PROMs were additionally documented to hospital-based recruitment of consecutive patients. The number of follow-ups varied, e.g. at 1 and 3 years, up to 5 years after diagnosis, or annually (without specified end date). Types of studies were observational prospective (or “cohort”) studies, and one included an interventional, sequential cohort approach (first control, then intervention). Questionnaires used were either not specified, or included EORTC-QLQ instruments or instruments specific to pediatric cancers (PedsQoL, TAPQOL).

      Sample characteristics

      The five programs included patients with various kinds of cancers. One included only prostate cancer patients, one included only bladder cancer patients, another included children up to 18 with all cancers and two included adult patients with different cancer types (breast, colon, rectum cancer and gastrointestinal, gynecological, neuroendocrine tumors, glioma, lung, and testicular cancer). Samples consisted of stage 0–III patients in one study, was not restricted in another and was not reported in three studies. The samples consisted of patients of all genders, except for the program on prostate cancer which was restricted to males. No restriction on ethnicity was indicated, but variation in ethnicity was also not reported. Sample sizes ranged from 158 to 22,332 patients.

      Qualitative synthesis of program characteristics

      Three of the five queried corresponding authors or their designated deputies responded. Two approved the syntheses [
      • Wintner L.M.
      • Giesinger J.M.
      • Zabernigg A.
      • et al.
      Evaluation of electronic patient-reported outcome assessment with cancer patients in the hospital and at home.
      ,
      • Zeegers M.P.
      • Bryan R.T.
      • Langford C.
      • et al.
      The West Midlands Bladder Cancer Prognosis Programme: rationale and design.
      ], with one [
      • Wintner L.M.
      • Giesinger J.M.
      • Zabernigg A.
      • et al.
      Evaluation of electronic patient-reported outcome assessment with cancer patients in the hospital and at home.
      ] adding information on items for which no information was provided in the study. Another study [
      • Engelen V.
      • Detmar S.
      • Koopman H.
      • et al.
      Reporting health-related quality of life scores to physicians during routine follow-up visits of pediatric oncology patients: is it effective?.
      ] provided comprehensive additional information which is documented in the supplementary material and synthesized in Table 3. Only one [
      • Engelen V.
      • Detmar S.
      • Koopman H.
      • et al.
      Reporting health-related quality of life scores to physicians during routine follow-up visits of pediatric oncology patients: is it effective?.
      ] filled out question 20 “Is there anything else that should be mentioned regarding the PROM program?”
      Few patterns were identified regarding the “Plan” items and information could often not be derived from the publications. The primary purposes of program implementation (in comparison to the research purposes of the articles described in Table 1), included the monitoring of events following treatment, identifying patients’ needs, benchmarking outcomes, improving communication between patients and providers or simply collecting quality of life information without further specification. Patient selection included full surveys of the whole (hospital) population affected as well as several strategies to approach patients during appointments. Regarding proxy support on the completion of surveys, three articles provided no information, one allowed parents to complete the questionnaires on behalf of their 0–7 year old or their mentally disabled children, and one explicitly mentioned that nurses could be asked for help. For two programs, information was given on why the specific questionnaires were selected (during the query), highlighting its psychometric properties and the available translations. Information on costs (items 6 and 7) were given sparsely, with two programs providing no details, one hinting at the necessity to provide a study nurse, one indicating that each participating unit had a specially-trained study nurse funded for the program and one recommending one PROM coordinator for each center. No cost estimates were given. Regarding recommendations for necessary conditions before implementation, the articles suggested staff training to use PROM instruments and the IT infrastructure as well as the provision of hardware for the on-site completion of questionnaires.
      Do” information regarding the implementation of the programs could be synthesized relatively well from the studies. PROMs implemented included cancer-generic instruments for studies with mixed populations and generic and cancer-type specific instruments for programs that included only one cancer type. The pediatric program tailored the selection of questionnaires according to patient age. PROM assessment was performed at very heterogeneous time points, for example “annually” or “prior to each consultation”. In two instances, no “baseline” PROMs were collected, as treatment-naïve patients were surveyed, but patients were surveyed in one program “within three months of diagnosis” [
      • Engelen V.
      • Detmar S.
      • Koopman H.
      • et al.
      Reporting health-related quality of life scores to physicians during routine follow-up visits of pediatric oncology patients: is it effective?.
      ] or the point of initial survey was not explicitly mentioned [
      • Meisner C.
      • Brinkmann F.
      • Henke D.
      • et al.
      [Benchmarking: how to measure outcome quality at the comprehensive oncology centre in Stuttgart].
      ]. Since these programs were nevertheless useful for monitoring patients and to investigate events following interventions these studies were considered eligible for the review. In two programs, questionnaires were completed electronically, in one instance via postal mailings and in one case via face-to face interviews. Assessment was done at home in some and exclusively at the time of consultation in other programs. Regarding data processing and result dissemination, three programs provide the results in a processed way to providers, either electronically or on paper. No information was found for the other two programs. PROMs were used for benchmarking in one program. The Austrian program stores data in hospitals separately, with no benchmarking effort to date. One program intends to use data for quality improvement purposes prospectively. For two programs, no information was found on benchmarking/provider comparison efforts. Two programs provided information on consequences of issues potentially identified with help of PROMs, with one discussing issues in workshops and one providing a decision-tree pocket card.
      In the “Study” dimension, we distinguished between the “barriers and enabler” and the “impact and lessons learned” aspect, with very little information being collected, primarily from the Austrian and Dutch programs. Critical issues identified for acceptance include internet and data privacy issues and sociodemographic barriers to electronic PROM collection as well as the inclusion of end-users in the development process. No information on how to promote multicentric implementation was identified. The Dutch program provided information on the impact of PROM implementation identifying better awareness of the providers regarding their patients’ health and a better provider-patient communication. Two aspects to be considered during implementation could be identified: the involvement of stakeholders including the training of patients may facilitate implementation and program initiators need to be aware that many difficulties may occur, for example that PROM collection may be more time-consuming than initially expected.
      The “Act” dimension inquired whether any modifications were done since program implementation with the only identified change in the Dutch program where – among other things – the infrastructure was changed to a fully web-based tool. The open question 20: “Is there anything else that should be mentioned regarding the PROM program?” was only answered by the Dutch respondent.

      Discussion

      Demanding the collection of PROMs to monitor patients or benchmark providers in routine cancer care has a long history [
      • Rose M.
      • Bezjak A.
      Logistics of collecting patient-reported outcomes (PROs) in clinical practice: an overview and practical examples.
      ]. Implementing such approaches, however, has proved to be difficult. Our systematic literature research and the consecutive survey of researchers involved in such approaches gave insight into the characteristics of existing programs as well as facilitators and barriers to implementation. Five programs from Europe were identified that met the inclusion criteria and allowed the collection of PROMs in routine cancer care to monitor (or detect changes) symptoms and function and that used identical data collection infrastructures across centers, to make the comparison of outcomes across providers possible, at least theoretically. In practice, it was often not clear whether and how the latter was done or planned at all, except for the example from Germany [
      • Meisner C.
      • Brinkmann F.
      • Henke D.
      • et al.
      [Benchmarking: how to measure outcome quality at the comprehensive oncology centre in Stuttgart].
      ] which had benchmarking as its focus. Ongoing approaches to provider comparison like the TNGR [
      • Evans S.M.
      • Millar J.L.
      • Moore C.M.
      • et al.
      Cohort profile: the TrueNTH Global Registry - an international registry to monitor and improve localised prostate cancer health outcomes.
      ,
      • Kowalski C.
      • Roth R.
      • Carl G.
      • et al.
      A multicenter paper-based and web-based system for collecting patient-reported outcome measures in patients undergoing local treatment for prostate cancer: first experiences.
      ] or EDIUM [
      • Breidenbach C.
      • Sibert N.T.
      • Wesselmann S.
      • et al.
      Das Potenzial von Patient-Reported Outcomes - Nutzen und Umsetzung der EDIUM-Studie.
      ] are too early in the process to be included in the review, and no results have been published at time of the database search. We suspect that there are more programs with data collection having started only recently and with no publications released so far, such as the ongoing PROM initiative in the 15 Tuscanian Breast centers
      https://www.santannapisa.it/it/ricerca/progetti/indagini-proms-nella-chirurgia-ricostruttiva-post-mastectomia
      , as well as some of the records identified in the search that were only conference abstracts and may be published as full articles in the near future. Another work that is worth mentioning is a Danish feasibility study that collected PROMs in lung cancer patients. This approach was not continued after the feasibility phase and only published shortly after our literature search. However, it may give valuable insight [
      • Bronserud M.M.
      • Iachina M.
      • Green A.
      • et al.
      Patient-reported outcomes (PROs) in lung cancer: Experiences from a nationwide feasibility study.
      ]. We emphasize that our aim was not to compare successful with unsuccessful approaches. Only “surviving” programs were included, i.e. programs for which at least one publication with routine data collection was released. In the absence of evidence from randomized trials regarding what works in PROM implementation in routine cancer care, we suggest that future reviews not only consider “surviving” programs but also those that failed to find their way into routine care and for what reasons.
      One of the purposes of this review was to give guidance to researchers and/or practitioners (preferably: both together) who want to implement PROM programs into practice. Although some of the studies identified gave valuable insight, many crucial aspects were less clear, particularly those regarding costs, necessary preconditions, how to handle the data collected, and how to allow for comparisons. Research and practice is at a still too early stage to give definite advice for implementation of PROM programs. We highlight some blind spots identified in this review that require further research:
      • 1.
        Resources/costs: In order to convince those in charge of budget allocation (e.g., funding institutions, hospital board members, policy makers, health insurances), it is imperative to identify costs of PROM collection in routine care, preferably on a per-patient basis. Ideally, these costs are compared to the resulting benefits. However, PROM collection is typically not covered by the payers (i.e. health insurances), which thwarts its implementation in routine care. We therefore want to encourage a discussion on whether routine PROM collection in cancer care should be financed by funders just like other interventions, and under which conditions.
      • 2.
        Prerequisites: Depending on the mode of data collection and the specific aims of PROM use, prerequisites may need to be fulfilled that require thorough advance planning. This may include IT infrastructure, the training of providers and acceptance building among those involved. Although the recent controlled PROM trials provide valuable arguments in favor of PROM use in routine care [
        • Basch E.
        • Deal A.M.
        • Dueck A.C.
        • et al.
        Overall Survival Results of a Trial Assessing Patient-Reported Outcomes for Symptom Monitoring During Routine Cancer Treatment.
        ,
        • Denis F.
        • Lethrosne C.
        • Pourel N.
        • et al.
        Randomized Trial Comparing a Web-Mediated Follow-up With Routine Surveillance in Lung Cancer Patients.
        ], this often means an additional workload for those involved that is not sufficiently reimbursed [
        • Schuler M.K.
        • Trautmann F.
        • Radloff M.
        • et al.
        Implementation of a mobile inpatient quality of life (QoL) assessment for oncology nursing.
        ,
        • Trautmann F.
        • Hentschel L.
        • Hornemann B.
        • et al.
        Electronic real-time assessment of patient-reported outcomes in routine care-first findings and experiences from the implementation in a comprehensive cancer center.
        ]. Especially for benchmarking/quality improvement purposes, a high response rate is imperative. This can only be achieved when the healthcare professionals see added value in routine PROM collection and fully support such programs.
      • 3.
        Presentation: In order to achieve provider commitment, data collected need to be processed in a way that is useful for clinical practice and fits into the workflow. This may involve the issue of how individual results are presented so that providers can use it in consultations and also when and how (paper, electronically) reports are provided [
        • Snyder C.F.
        • Smith K.C.
        • Bantug E.T.
        • et al.
        What do these scores mean? Presenting patient-reported outcomes data to patients and clinicians to improve interpretability.
        ]. Ideally, patient-individual reports will be part of the overall electronic infrastructure of the providing institution and, as such, implemented into the clinical pathway. As obvious as this idea is, such an approach is apparently easier said than done. The literature on this issue typically stems from randomized trials or qualitative research and not from programs that allow for a comparison in routine care [
        • Boyce M.B.
        • Browne J.P.
        • Greenhalgh J.
        The experiences of professionals with using information from patient-reported outcome measures to improve the quality of healthcare: a systematic review of qualitative research.
        ].
      • 4.
        Response: Connected to the issue of how results are presented is the question of how to react to the results. This includes the question of whom to include in the discussion of results, how to identify areas of intervention, and what to do once areas of interventions are identified, typically when impaired functioning is detected. This aspect is heavily connected to how reports are presented, e.g. if impaired functioning is flagged, and bears several methodological and ethical challenges. For example, how can thresholds for impairment that require an intervention be defined? Or, if many dimensions of functioning are impaired, should all be “treated” or only the most severe? Klinkhammer-Schalke and colleagues suggested a multidisciplinary approach to discuss results and plan interventions [
        • Klinkhammer-Schalke M.
        • Koller M.
        • Steinger B.
        • et al.
        Direct improvement of quality of life using a tailored quality of life diagnosis and therapy pathway: randomised trial in 200 women with breast cancer.
        ]. We want to stress the importance of the inclusion of not only physicians but all relevant disciplines, among them nurses, social workers, and psycho-oncologists. PROM collection and analysis does not necessarily center around physicians but can be led by other professions as well [
        • Kotronoulas G.
        • Papadopoulou C.
        • MacNicol L.
        • et al.
        Feasibility and acceptability of the use of patient-reported outcome measures (PROMs) in the delivery of nurse-led supportive care to people with colorectal cancer.
        ]. Whether multidisciplinary, comprehensive approaches like [
        • Klinkhammer-Schalke M.
        • Koller M.
        • Steinger B.
        • et al.
        Direct improvement of quality of life using a tailored quality of life diagnosis and therapy pathway: randomised trial in 200 women with breast cancer.
        ] can be implemented into routine care certainly depends on the resources a health care system has.
      • 5.
        Multicentricity: The issue of both presentation and response is a blind spot for the comparison of results across providers. There is little resistance to the idea that fair comparisons require case mix (or: risk) adjustment including baseline (pre-intervention) PROM data. However, what models and what co-variates should be used for adjustment is less clear, but may make a big difference. Also, how data from different providers are transferred to one database may prove difficult when not planned in advance. Only one program had benchmarking as an explicit aim and we argue that (patient-reported) outcome comparison is critical for quality improvement.
      Our systematic review was based on a sensitive search string to identify programs for PROM collection that may be “hidden” in the literature, for example when they are not presented as “programs”, but merely provide the basis for controlled trials in which PROMs are collected, for example as secondary outcomes. A supplementary hand search and a query among colleagues were conducted to identify programs not identified with the systematic search. The sensitivity of the search string led to a high number of excluded records at the title/abstract screening stage. A substantial share of otherwise promising records identified were only published as conference abstracts at the time of the database search. This suggests that there are several programs ongoing that are not published as full papers yet. The implementation of PROMs in routine care is a relatively new approach and it is likely we will see many more such programs in the near future. However, we cannot rule out having missed relevant programs, also because of terminology issues: “Programs” that are part of routine care may be difficult to distinguish from “studies” or “projects”. Both may be planned for a limited time, like the three years typically allotted for research, but also with a longer time perspective without being labelled as such. Also, it is often not clear at program initiation how long such an initiative will last.
      Insight into “what works” in cancer routine care about PROM implementation may benefit from other areas of healthcare that are beyond the scope of this review. This includes work on surgical procedures, as shown by the UK or Swedish examples mentioned above. Methodological papers based on these programs include the role of patient characteristics on differential recruitment rates or the significance of PROMs for outcome comparisons as compared to mortality [
      • Hutchings A.
      • Neuburger J.
      • van der Meulen J.
      • et al.
      Estimating recruitment rates for routine use of patient reported outcome measures and the impact on provider comparisons.
      ,
      • Varagunam M.
      • Hutchings A.
      • Black N.
      Do patient-reported outcomes offer a more sensitive method for comparing the outcomes of consultants than mortality?. A multilevel analysis of routine data.
      ]. Looking beyond Europe allows for the identification of further important methodological aspects like different procedures for risk-adjustment [
      • Burgess R.
      • Bishop A.
      • Lewis M.
      • et al.
      Models used for case-mix adjustment of patient reported outcome measures (PROMs) in musculoskeletal healthcare: A systematic review of the literature.
      ]. However, since cancer care with its necessarily multidisciplinary approach differs substantially from many other fields of medicine, we must be careful when adopting conclusions from other fields.

      Funding

      This report arises from the Innovative Partnership for Action Against Cancer Joint Action (IPAAC), which has received funding from the European Union through the Consumers, Health, Agriculture and Food Executive Agency of the European Commission, in the framework of the Health Programme 2014-2020. The European Commission is not responsible for the content of this report. The sole responsibility for the report lies with the authors, and the Consumers, Health, Agriculture and Food Executive Agency is not responsible for any use that may be made of the information contained herein. The authors are not responsible for any further and future use of the report by third parties and third-party translations.

      Acknowledgments

      The authors would like to thank Giuseppe La Torre, Nikolai Goncharenko, and Sara Perreira for their valuable input as well as the members of work package 10, task 4 of the IPAAC Joint Action.

      Conflict of interest

      The authors declare no conflict of interest in relation to this work.

      References

      1. OECD. RECOMMENDATIONS TO OECD MINISTERS OF HEALTH FROM THE HIGH LEVEL REFLECTION GROUP ON THE FUTURE OF HEALTH STATISTICS. Strengthening the international comparison of health system performance through patient-reported indicators; 2017.

        • Efficace F.
        • Rosti G.
        • Aaronson N.
        • et al.
        Patient- versus physician-reporting of symptoms and health status in chronic myeloid leukemia.
        Haematologica. 2014; 99: 788-793
        • Fromme E.K.
        • Eilers K.M.
        • Mori M.
        • et al.
        How accurate is clinician reporting of chemotherapy adverse effects?.
        A comparison with patient-reported symptoms from the Quality-of-Life Questionnaire C30, Journal of clinical oncology: official journal of the American Society of Clinical Oncology. 2004; 22: 3485-3490
        • Basch E.
        • Abernethy A.P.
        Supporting Clinical Practice Decisions With Real-Time Patient-Reported Outcomes.
        Journal of Clinical Oncology. 2011; 29: 954-956
        • Klinkhammer-Schalke M.
        • Koller M.
        • Steinger B.
        • et al.
        Direct improvement of quality of life using a tailored quality of life diagnosis and therapy pathway: randomised trial in 200 women with breast cancer.
        British journal of cancer. 2012; 106: 826-838
        • Basch E.
        • Deal A.M.
        • Dueck A.C.
        • et al.
        Overall Survival Results of a Trial Assessing Patient-Reported Outcomes for Symptom Monitoring During Routine Cancer Treatment.
        JAMA. 2017; 318: 197-198
        • Denis F.
        • Lethrosne C.
        • Pourel N.
        • et al.
        Randomized Trial Comparing a Web-Mediated Follow-up With Routine Surveillance in Lung Cancer Patients.
        J Natl Cancer Inst. 2017; : 109
        • Snyder C.F.
        • Aaronson N.K.
        • Choucair A.K.
        • et al.
        Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations.
        Quality of life research: an international journal of quality of life aspects of treatment, care and rehabilitation. 2012; 21: 1305-1314
        • Zerillo J.A.
        • Schouwenburg M.G.
        • van Bommel A.C.M.
        • et al.
        An International Collaborative Standardizing a Comprehensive Patient-Centered Outcomes Measurement Set for Colorectal Cancer.
        JAMA Oncol. 2017; 3: 686-694
        • Devlin N.J.
        • Appleby J.
        Getting the most out of PROMs Putting health outcomes at the heart of NHS decision-making. In..
        The King's Fund;, London2010
        • Rolfson O.
        • Karrholm J.
        • Dahlberg L.E.
        • et al.
        Patient-reported outcomes in the Swedish Hip Arthroplasty Register: results of a nationwide prospective observational study.
        The Journal of bone and joint surgery British volume. 2011; 93: 867-875
        • Evans S.M.
        • Millar J.L.
        • Moore C.M.
        • et al.
        Cohort profile: the TrueNTH Global Registry - an international registry to monitor and improve localised prostate cancer health outcomes.
        BMJ open. 2017; 7: e017006
        • Wintner L.M.
        • Sztankay M.
        • Aaronson N.
        • et al.
        The use of EORTC measures in daily clinical practice—A synopsis of a newly developed manual.
        European Journal of Cancer. 2016; 68: 73-81
        • Rose M.
        • Bezjak A.
        Logistics of collecting patient-reported outcomes (PROs) in clinical practice: an overview and practical examples.
        Qual Life Res. 2009; 18: 125-136
      2. International Society for Quality of Life Research (prepared by Aaronson, N., E.T., Greenhalgh, J., Halyard, M., Hess, R., Miller, D., Reeve, B., Santana, M., Snyder, C)., User's Guide to Implementing Patient-Reported Outcomes Assessment in Clinical, Practice., In; 2015.

        • Franklin P.
        • Chenok K.
        • Lavalee D.
        • et al.
        Framework To Guide The Collection And Use Of Patient-Reported Outcome Measures In The Learning Healthcare System.
        EGEMS (Wash DC). 2017; 5: 17
        • Nordan L.
        • Blanchfield L.
        • Niazi S.
        • et al.
        Implementing electronic patient-reported outcomes measurements: challenges and success factors.
        BMJ Quality & Safety. 2018; (DOI: 10.1136/bmjqs-2018-008426)
      3. iPAAC Consortium. Innovative Partnership for Action against Cancer (iPAAC) - Official iPAAC Joint Action presentation. In; 2018.

        • Moher D.
        • Liberati A.
        • Tetzlaff J.
        • et al.
        Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
        PLoS Med. 2009; 6: e1000097
        • Engelen V.
        • Detmar S.
        • Koopman H.
        • et al.
        Reporting health-related quality of life scores to physicians during routine follow-up visits of pediatric oncology patients: is it effective?.
        Pediatric blood & cancer. 2012; 58: 766-774
        • Meisner C.
        • Brinkmann F.
        • Henke D.
        • et al.
        [Benchmarking: how to measure outcome quality at the comprehensive oncology centre in Stuttgart].
        Z Evid Fortbild Qual Gesundhwes. 2011; 105: 365-370
        • Nguyen-Nielsen M.
        • Hoyer S.
        • Friis S.
        • et al.
        The Danish Prostate Cancer Database.
        Clinical epidemiology. 2016; 8: 649-653
        • Wintner L.M.
        • Giesinger J.M.
        • Zabernigg A.
        • et al.
        Evaluation of electronic patient-reported outcome assessment with cancer patients in the hospital and at home.
        BMC Med Inform Decis Mak. 2015; 15: 110
        • Zeegers M.P.
        • Bryan R.T.
        • Langford C.
        • et al.
        The West Midlands Bladder Cancer Prognosis Programme: rationale and design.
        BJU Int. 2010; 105: 784-788
        • Kowalski C.
        • Roth R.
        • Carl G.
        • et al.
        A multicenter paper-based and web-based system for collecting patient-reported outcome measures in patients undergoing local treatment for prostate cancer: first experiences.
        J Patient Rep Outcomes. 2020; 4: 56
        • Breidenbach C.
        • Sibert N.T.
        • Wesselmann S.
        • et al.
        Das Potenzial von Patient-Reported Outcomes - Nutzen und Umsetzung der EDIUM-Studie.
        Onkologische Pflege. 2019; (DOI: 10.4486/j.op.2019.02.08:): 50-53
        • Bronserud M.M.
        • Iachina M.
        • Green A.
        • et al.
        Patient-reported outcomes (PROs) in lung cancer: Experiences from a nationwide feasibility study.
        Lung cancer (Amsterdam, Netherlands). 2019; 128: 67-73
        • Schuler M.K.
        • Trautmann F.
        • Radloff M.
        • et al.
        Implementation of a mobile inpatient quality of life (QoL) assessment for oncology nursing.
        Supportive care in cancer: official journal of the Multinational Association of Supportive Care in Cancer. 2016; 24: 3391-3399
        • Trautmann F.
        • Hentschel L.
        • Hornemann B.
        • et al.
        Electronic real-time assessment of patient-reported outcomes in routine care-first findings and experiences from the implementation in a comprehensive cancer center.
        Supportive care in cancer: official journal of the Multinational Association of Supportive Care in Cancer. 2016; 24: 3047-3056
        • Snyder C.F.
        • Smith K.C.
        • Bantug E.T.
        • et al.
        What do these scores mean? Presenting patient-reported outcomes data to patients and clinicians to improve interpretability.
        Cancer. 2017; (DOI: 10.1002/cncr.30530: n/a-n/a)
        • Boyce M.B.
        • Browne J.P.
        • Greenhalgh J.
        The experiences of professionals with using information from patient-reported outcome measures to improve the quality of healthcare: a systematic review of qualitative research.
        BMJ Quality & Safety. 2014; 23: 508-518
        • Kotronoulas G.
        • Papadopoulou C.
        • MacNicol L.
        • et al.
        Feasibility and acceptability of the use of patient-reported outcome measures (PROMs) in the delivery of nurse-led supportive care to people with colorectal cancer.
        European Journal of Oncology Nursing. 2017; 29: 115-124
        • Hutchings A.
        • Neuburger J.
        • van der Meulen J.
        • et al.
        Estimating recruitment rates for routine use of patient reported outcome measures and the impact on provider comparisons.
        BMC health services research. 2014; 14: 66
        • Varagunam M.
        • Hutchings A.
        • Black N.
        Do patient-reported outcomes offer a more sensitive method for comparing the outcomes of consultants than mortality?. A multilevel analysis of routine data.
        BMJ Quality & Safety. 2015; 24: 195-202
        • Burgess R.
        • Bishop A.
        • Lewis M.
        • et al.
        Models used for case-mix adjustment of patient reported outcome measures (PROMs) in musculoskeletal healthcare: A systematic review of the literature.
        Physiotherapy. 2019; 105: 137-146
        • Santana M.J.
        • Haverman L.
        • Absolom K.
        • et al.
        Training clinicians in how to use patient-reported outcome measures in routine clinical practice.
        Qual Life Res. 2015; 24: 1707-1718
        • Haverman L.
        • van Oers H.A.
        • Limperg P.F.
        • et al.
        Implementation of electronic patient reported outcomes in pediatric daily clinical practice: The KLIK experience.
        Clinical Practice in Pediatric Psychology. 2014; 2: 50-67
        • Engelen V.
        • Haverman L.
        • Koopman H.
        • et al.
        Development and implementation of a patient reported outcome intervention (QLIC-ON PROfile) in clinical paediatric oncology practice.
        Patient education and counseling. 2010; 81: 235-244
        • Schepers S.A.
        • Sint Nicolaas S.M.
        • Haverman L.
        • et al.
        Real-world implementation of electronic patient-reported outcomes in outpatient pediatric cancer care.
        Psychooncology. 2017; 26: 951-959
        • Schepers S.A.
        Changing pediatric cancer care: development and implementation of electronic patient and parent reported outcomes.
        Gildeprint, Amsterdam2017
        • Engelen V.
        • Detmar S.
        • Koopman H.
        • et al.
        Reporting health-related quality of life scores to physicians during routine follow-up visits of pediatric oncology patients: Is it effective?.
        PediatrBlood Cancer. 2012; 58: 766-774
        • Haverman L.
        Electronic patient and parent reported outcomes in pediatric clinical practice. In.
        University of Amsterdam UvA-DARE (Digital Academic Repository), Amsterdam2013