Summary
Background
The National Association of Statutory Health Insurance Physicians develops quality indicators (QIs) for ambulatory care in Germany. This study explores the feasibility of a total of 48 QIs.
Methods
Cross-sectional observational study with primary data collection in writing from medical practices in 10 specialist fields of outpatient care. “Feasibility” covers 7 criteria for indicator assessment and data collection: applicability, availability, retrievability, complexity, relevance, reliability, and acceptance. A questionnaire consisting of 10 questions was used to evaluate these feasibility criteria for each indicator. Survey results were subjected to descriptive analysis.
Results
The analyzed sample comprises 103 participants who have been working as practice-based physicians for an average of 13 years. 40% only keep electronic medical records and 2% only paper records, and the rest uses both.
The rating of QIs in the field-specific QI sets shows the following mean values: 67% of the participants consider the QIs assigned to them as corresponding to their practice care mandate. Data on these QIs deemed to be applicable are collected by 94% of respondents, documented by 91%, and by 51% electronically. 58% of the data required for the denominator, and 38% for the numerator are retrievable from the practice management system. The time required to access data on a QI is more than 30 minutes for 84% of respondents, and 67% consider the effort involved as unacceptable. The rating received was 61% for the relevance of QIs to the assessment of a practitioner's own quality of health care, 69% for the estimated reliability of data collection, and 58% for the acceptance of being evaluated via QIs.
Conclusions
In order to improve the feasibility of QI-based practice assessments it will be necessary to a) fine-tune the selection of QIs for the respective groups of specialist, b) to promote the use of computerized practice management systems, and c) integrate effective and user-friendly retrieval functions in the software. Another aspect to be explored is how the acceptance of QI-based practice evaluations can be improved in individual specialist fields.
Zusammenfassung
Hintergrund
Die Kassenärztliche Bundesvereinigung (KBV) entwickelt Qualitätsindikatoren (QI) für die ambulante ärztliche Versorgung in Deutschland. Die vorliegende Untersuchung überprüft die Machbarkeit von insgesamt 48 QI, mit denen die Qualität der ambulanten Versorgung in 10 Fachgruppen (FG) deutscher Arztpraxen bewertet werden soll.
Methode
Ambulant tätige Ärzte einer Gelegenheitsstichprobe, geschichtet in 10 FG, wurden schriftlich zu ihrer Einschätzung der Machbarkeit der Indikatoren befragt. Jede FG bekam zwischen 10 und 46 QI in ihrem FG-QI-Satz zur Bewertung vorgelegt. Die Zusammenstellung der 10 FA-QI-Sätze erfolgte in einem Konsensusverfahren durch eine Expertengruppe der KBV gemäß Versorgungsauftrag jeder FG. „Machbarkeit“ umfasst in dieser Untersuchung 7 Kriterien zur Indikatorbewertung und Datenerhebung: Anwendbarkeit, Verfügbarkeit, Abrufbarkeit, Aufwand, Relevanz, Zuverlässigkeit und Akzeptanz. Als Erhebungsinstrument diente ein selbst entwickelter, 10 Fragen umfassender Fragebogen zur Bewertung der 7 Machbarkeitskriterien eines jeden Indikators. Die Befragungsergebnisse wurden deskriptiv analysiert.
Ergebnisse
Die Stichprobe besteht aus 103 Teilnehmern, die im Mittel 51 Jahre alt, zu 17% weiblich und seit 13 Jahren niedergelassen sind, davon 44% in Einzelpraxen, 56% in Mehrbehandlerpraxen. 40% führen Patientenakten nur elektronisch, 2% nur in Papierform, der Rest verwendet beide Dokumentationsformen.
Die Bewertung der QI in den FG-QI-Sätzen zeigt folgende Mittelwerte: 67% der Praxen in den FG halten die ihnen zugeordneten QI für zutreffend für den Versorgungsauftrag ihrer Praxis. Die Informationen für diese als zutreffend eingeschätzten QI werden zu 94% erhoben, zu 91% dokumentiert, dabei zu 51% rechnergestützt. Abrufbar sind 58% der notwendigen Nenner- und 38% der Zählerinformationen. Der Aufwand, um die Daten für einen QI abzurufen, beträgt bei 84% mehr als 30 Minuten und 67% bewerten den Aufwand als nicht zumutbar. Die Relevanz der QI im Hinblick auf die Bewertung der Versorgungsqualität der eigenen Arbeit liegt bei 61%, die eingeschätzte Zuverlässigkeit der Erhebung bei 69% und die Akzeptanz, mit dem QI bewertet zu werden, bei 58%.
Schlussfolgerungen
Um die Machbarkeit QI-basierter Praxisbewertungen in deutschen Praxen zu erhöhen, sollte a) die Passung der QI zu den einzelnen FG verbessert werden, b) die Verwendung EDV-basierter Praxisverwaltungssysteme gefördert werden und c) sollten unterstützende, nutzerfreundliche Abruffunktionen in die Praxisverwaltungsprogramme integriert werden. Darüber hinaus ist zu erforschen, wie die Akzeptanz QI-basierter Praxisbewertungen in einzelnen Fachgruppen gefördert werden kann.
Keywords
Schlüsselwörter
Background
Quality is a core target criterion in health care [
[1]
], and quality indicators (QIs) play a crucial role in quality assessment as a tool to map aspects of the quality of care. The development of quality indicators must satisfy criteria such as relevance, scientific validity and feasibility, and preferably be substantiated by sound studies [2
, 3
, 4
, - Lüngen M.
- Rath T.
Analyse und Evaluierung des QUALIFY Instruments zur Bewertung von Qualitätsindikatoren anhand eines strukturierten qualitativen Interviews. [Analysis and evaluation of the QUALIFY tool for assessing quality indicators with structured group interviews.].
Z Evid Fortbild Qual Gesundhwes. 2011; 105: 38-43
5
, 6
]. Feasibility is essential for the implementation of an indicator-based system and one of the core criteria against which NICE independent advisory committees test all potential indicators [[7]
].Quality indicators have been increasingly developed, introduced and applied in outpatient care at an international level over the past years [
3
, 8
, 9
, 10
]. However, the science of quality indicator development and implementation still leaves much room for improvement, especially in Germany [11
, - Schmitt J.
- Petzold T.
- Eberlein-Gonska M.
- Neugebauer E.A.M.
Anforderungsprofil an Qualitätsindikatoren. Relevanz aktueller Entwicklungen der Outcomes Forschung für das Qualitätsmanagement [Requirements for quality indicators. The relevance of current developments in outcomes research for quality management].
Z Evid Fortbild Qual Gesundhwes. 2013; 107: 516-522
12
, 13
, 14
, 15
].In the German health care system the National Association of Statutory Health Insurance Physicians (NASHIP) aims to establish a quality-based performance assessment. This concept requires QIs adapted to the German health care system and data to calculate the QIs. To achieve this goal, the NASHIP launched a project to develop ambulatory quality indicators and key measures (AQUIK) in 2006. The project covers 3 phases [
[16]
]. In phase 1 a QI search was conducted and in phase 2 an expert panel selected 48 QIs from which 10 different specialist groups QI sets were assembled applying an adapted UCLA-RAND technique. It was decided not to include any QIs referring to the 6 diagnoses of the German DMPs (diabetes mellitus 1 and 2, asthma, chronic obstructive pulmonary disease, coronary heart disease, and breast cancer) since QI are already available [- Kleudgen S.
- Diel F.
- Burgdorf F.
- Quasdorf I.
- de Cruppé W.
- Geraedts M.
KBV entwickelt Starter-Set ambulanter Qualitätsindikatoren - AQUIK®-Set. [AQUIK®: Starter set of ambulatory quality indicators developed by the German National Association of Statutory Health Insurance Physicians.].
Z Evid Fortbild Qual Gesundhwes. 2011; 105: 54-63
[17]
]. Phase 3 consists of a feasibility study which is described in this article.Data availability in the German system of ambulatory care
Nowadays, almost all practices use Electronic Data Processing (EDP) for billing. And there is a growing number of practices that use electronic medical records (EMR). According to the Commonwealth Fund surveys 42% of German GPs reported to use EMR in 2006, 72% in 2009, and 82% in 2012. However in 2012, only 7% reported an EDP system with “multifunctional electronic health information capacity” [
18
, 19
, 20
]. Therefore, there are no routine data records available for all German outpatient practices with practice identification, on clinical care, on practice structures and process data. And there is no comprehensive information available on how clinical data are recorded and retrievable. In order to implement a practice-based quality assessment the required data must be collected in each single medical practice. Against this background, phase 3 of the AQUIK project attended to study the feasability of practice-based quality assessment in Germany's ambulatory care setting.Methods
Study design
This is a cross sectional observational study with primary data collection in writing from medical practices in 10 specialist fields of outpatient care: gynaecology, general medicine, otolaryngology, internal medicine, paediatrics, child and youth psychiatry, neurology/psychiatry, orthopaedics, HIV specialization, urology.
Study materials
48 quality indicators, developed in phase 1 and 2 of the AQUIK project [
[17]
], arranged in 10 sets of field-specific quality indicators are tested for feasibility. The 48 quality indicators are divided into 9 thematic areas (see left column in tab. 2). “Practice management” comprises 5 generic indicators on practice structures and processes that apply to all specialist groups and form part of all quality indicator sets. “Multidisciplinary topics” with three general preventive indicators on blood pressure and smoking, also “Drug safety” and “Immunisation” are contained in many sets. The remaining 4 areas cover clinical issues and are assigned to only one or few fields respectively. The 48 quality indicators therefore appear in various distributions and combinations in the 10 specialist fields. All 48 quality indicators are uniformly structured and described in detail as recommended by the Joint Commission on Accreditation of Health Care Organizations [[21]
]. Each indicator information is available in a long and a short version. This feasibility study uses the short versions following the uniform structure: 1. indicator definition, 2. definition of numerator and denominator, 3. explanation of indicator, 4. type of indicator, 5. data compilation for indicator with data compilation of numerator and denominator. All detailed information on each quality indicator can be found at the NASHIP homepage [[17]
]. The 5 quality indicators in the section “practice management” determine as numerator the existence of a practice structure or implementation of a practice process, based on the denominator “private practice”, to be answered by yes or no. All 43 quality indicators of the other 8 sections determine rates and require practice data on numerator and denominator.Definition of feasibility
The process of implementing QIs comprises several steps [
[22]
]. It is essential to determine the feasibility of the developed QIs and take the findings into consideration. Internationally, the term ‘feasibility’ covers a variety of different aspects; there is no standardised definition nor validated survey instrument. But in general, feasibility refers to practical measures of implementation, adapted to the respective study, stage of implementation and application context [3
, 5
, 8
, 23
, 24
, 25
, 26
, 27
]. The present study applies 7 different criteria which in their combination illustrate the feasibility of a QI:- 1.applicability of the QI in the respective medical practice
- 2.availability of the data required for the QI in the respective medical practice
- 3.retrievability of QI data
- 4.complexity in retrieving QI data
- 5.relevance of QI for patient care
- 6.estimation of reliability of QI data compilation
- 7.doctors’ acceptance of QI for evaluation
The study is no trial data compilation of data for QIs from the practice management systems, nor does it analyse the QI data for completeness, accuracy, consistency, validity and reliability.
Study instrument
A specifically designed written questionnaire is used for the survey. It comprises a practice structure form with 17 items on the medical practice and sociodemography of respondents and an indicator assessment form with 10 questions. Both parts of the assessment tool are identical for all practices. An indicator assessment form addresses the assessment of one quality indicator. This is why the questionnaire for each specialist group respectively covers as many indicator assessment forms as the number of quality indicators in the field-specific set.
The 7 feasibility criteria are operationalized in the indicator questionnaire in such a way that the sequence of the 10 questions plausibly reflects the following steps: rating if quality indicator applies to the respective medical practice, collecting data, documenting data, having data available, and evaluating quality indicators. The first item in the indicator questionnaire (see Appendix A) asks for the basic applicability of an indicator (question 1) to prove that this indicator is in fact part of the daily routine in this practice. Thereby assuring that physicians assessed the QI only out of their own practical experience. Question 2a asks whether data of this quality indicator is actually collected. These two items are summed up under the heading “applicability”. The criterion “availability” inquires whether collected data on a quality indicator are documented as well (2b), and in which format, i.e. electronically or on paper (3). Access to documented information is covered by the third criterion “retrievability” (questions 4 and 5). The criterion “complexity” reflects the effort involved in retrieving data under current practice conditions (6), and whether this effort is reasonable (7). The last three questions concern the practitioner's assessment of the quality indicator and its relevance with regard to professional activities (8), the reliability of data collection from practice records (9), and the acceptance of the quality indicator as a tool to assess the quality of care in a medical practice of the respective field (10). Questionnaires and instructions how to complete them underwent two preliminary tests with 3 and 5 practices respectively.
Study conduct
115 medical practices who responded to a public call from the NASHIP received the questionnaire in November 2008 by post. After a 4-week period written reminders were sent out to the medical practices with a two-week extension of the deadline after which participation was clarified by phone. All participating medical practices received € 200 in compensation for expenses. Data were entered into MS Office Excel®. A double entry of one questionnaire of each specialist group (two of GPs) determined the rate of entry errors, which is 0.03%.
Statistical analyses
The SPSS® software was used for descriptive statistical analysis. Indicator assessment forms were evaluated for each quality indicator accross all specialist fields. Answers from all participating practices for which the QI was included in their field-specific set were evaluated together for this purpose. On the other hand, results of the 10 questions for each specialist field were evaluated together across all QIs of the field-specific sets. Question 1 whether the respondent believes the QI to be relevant for the medical practice is a filter question for both formats of result presentation, i.e. according to QI and to specialist field. The subsequent calculation of indicator questions 2 to 10 is restricted to respondents who qualify the indicator as applying. QIs which no respondent in a specialist field rated as applying to his medical practice were no longer considered in the percentage calculation of assessment per QI and specialist field. Calculations were based on dichotomised answer categories which differentiate between answers “positive/correct” and “negative/not applicable”.
Results
Study sample
The sample is not representative, and drawn from mainly 10 medical practices of each specialist field (20 from GPs and 3 HIV specialists). It constitutes a convenience sample stratified according to specialist groups. 103 out of 115 practices returned the completed questionnaire (response rate 90%) and thus form the study sample (100%). 9 specialist groups were evaluated in the assessment according to groups; the group of HIV specialists was not taken into account since only one practice participated in the study.
Sociodemographic and structural practice characteristics are listed in Tab. 1. Compared to the NASHIP statistics for all medical practices the percentage of female physicians in medical practices fluctuates between 7 and 64% depending on the specialist field. Female physicians are therefore underrepresented in this study. The average age determined in the study, however, corresponds exactly to the statistical average age. According to NASHIP figures, 35% of all 138.300 physicians in ambulatory care work in joint or group practices, and 4% in MCUs. No data are available on the proportion of practices with a certified quality management system in Germany, but studies indicate [
[28]
] a rate far below the 72% found for this sample. Much of the electronically documented clinical information is entered in the practice IT system in the form of free text fields, only rarely in predefined and therefore easily retrievable text fields. According to the physicians’ level of knowledge, the IT systems in all medical practices offer a registry function to call up statistical data. Most practices make use of this function in some way (87%). Again according to the physicians’ level of knowledge, between half and three quarters of IT systems used offer additional functions such as reminder or warning functions; these are used by one third to three quarters.Table 1Sociodemography and practice structures of the study sample.
| sociodemography and practice structures | participants |
|---|---|
| Age (average) | 51 years |
| female | 17% |
| practicing years (average) | 13 years |
| single practice | 44% |
| group practice or joint practice | 48% |
| medical care unit | 8% |
| number of medical and administrative staff (average) | 7 persons |
| number of patients per quarter (average) | 1500 |
| patients with statutory health insurance (average) | 89% |
| patients with private health insurance (average) | 11% |
| participation in disease management programs | 43% |
| quality management certificate | 72% |
| electronic medical records only | 40% |
| paper medical records only | 2% |
| electronic and paper medical records | 58% |
Results of indicator assessment
Results of indicator assessments are presented from two different perspectives. Results of the 10 questions on each of the 48 quality indicators across all respondents who completed the QI assessment questionnaire are given first, followed by a presentation of results according to field-specific QI sets. The structure of result presentation corresponds to the 7 feasibility criteria respectively.
Assessment of each QI across all surveyed specialist groups
Tab. 2 shows evaluation results of the 10 indicator questions for each of the 48 QIs across all participating medical practices whose field-specific QI set contains the indicator and who rated it as applying to their practice. The table shows the mean value of reported positive or approving assessments for each indicator question.
Table 2Assessment of each indicator across all specialist groups.
| QI No | QI short description | number surveyed practices | number practices QI applies (question 1) | % practices QI applies (question 1) | % data collected (question 2a) | % data documented (question 2b) | % data computerized (question 3) | % denominator retrievable (question 4) | % numerator retrievable (question 5) | % <30 minutes required (question 6) | % reasonable amount of time (question 7) | % relevant (question 8) | % reliable (question 9) | % acceptable (question 10) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Practice management | ||||||||||||||
| 1 | home visits | 103 | 60 | 59 | no practice data documentation necessary | 57 | 73 | 53 | ||||||
| 2 | drug allergies | 103 | 90 | 87 | 78 | 80 | 71 | |||||||
| 3 | reviews of significant events | 103 | 91 | 88 | 95 | 78 | 85 | |||||||
| 4 | emergency drugs | 103 | 92 | 89 | 91 | 87 | 86 | |||||||
| 5 | training/updating | 103 | 90 | 87 | 84 | 91 | 84 | |||||||
| Multidisciplinary topics | ||||||||||||||
| 6 | blood pressure measurement | 52 | 41 | 79 | 100 | 100 | 59 | 54 | 49 | 25 | 18 | 60 | 75 | 56 |
| 7 | tobacco use | 103 | 62 | 60 | 98 | 98 | 48 | 46 | 26 | 27 | 22 | 43 | 58 | 37 |
| 8 | smoking cessation | 83 | 33 | 40 | 96 | 92 | 55 | 30 | 18 | 10 | 10 | 34 | 39 | 31 |
| Drug safety | ||||||||||||||
| 9 | long-term medication | 82 | 58 | 71 | 96 | 91 | 61 | 37 | 32 | 11 | 11 | 54 | 63 | 54 |
| 10 | oral anticoagulation | 33 | 27 | 82 | 98 | 96 | 63 | 67 | 52 | 41 | 42 | 63 | 81 | 67 |
| Immunisation | ||||||||||||||
| 11 | influenza vaccine | 82 | 34 | 41 | 84 | 71 | 71 | 65 | 41 | 38 | 33 | 45 | 67 | 55 |
| 12 | tetanus and diphtheria | 53 | 36 | 68 | 97 | 94 | 63 | 72 | 53 | 28 | 42 | 61 | 78 | 53 |
| 13 | infant immunisation status | 44 | 19 | 43 | 95 | 95 | 63 | 74 | 58 | 21 | 42 | 74 | 74 | 68 |
| 14 | adolescent immunisation status | 53 | 35 | 66 | 92 | 93 | 51 | 63 | 40 | 20 | 23 | 63 | 62 | 57 |
| Gynaecological indicators, urinary incontinence, AIDS/HIV | ||||||||||||||
| 15 | pregnancy/smoking cessation | 41 | 11 | 27 | 100 | 96 | 45 | 45 | 27 | 18 | 45 | 55 | 64 | 55 |
| 16 | cervical screening/follow-up | 9 | 9 | 100 | 100 | 100 | 56 | 56 | 33 | 33 | 56 | 89 | 100 | 100 |
| 17 | sexually transmitted diseases/counselling | 9 | 8 | 89 | 100 | 100 | 50 | 38 | 13 | 13 | 25 | 50 | 63 | 50 |
| 18 | urinary incontinence – differential diagnosis | 51 | 35 | 69 | 95 | 93 | 68 | 59 | 29 | 29 | 26 | 62 | 65 | 50 |
| 19 | urinary incontinence – treatment options | 51 | 33 | 65 | 95 | 93 | 67 | 55 | 27 | 24 | 24 | 52 | 61 | 45 |
| 20 | AIDS/HIV – test viral load | 63 | 3 | 5 | 67 | 67 | 67 | 33 | 0 | 33 | 33 | 33 | 33 | 33 |
| 21 | AIDS/HIV – reduction viral load | 63 | 5 | 8 | 67 | 83 | 67 | 0 | 0 | 33 | 33 | 33 | 33 | 33 |
| 22 | AIDS/HIV – hepatitis C status | 52 | 7 | 13 | 78 | 78 | 57 | 0 | 0 | 29 | 14 | 29 | 29 | 29 |
| Cardiovascular diseases | ||||||||||||||
| 23 | blood pressure monitoring | 32 | 29 | 90 | 100 | 100 | 66 | 66 | 52 | 32 | 32 | 86 | 93 | 82 |
| 24 | arterial hypertension – education about risk factors | 32 | 29 | 90 | 100 | 99 | 55 | 66 | 14 | 11 | 11 | 71 | 64 | 61 |
| 25 | arterial hypertension – plan of care | 32 | 26 | 81 | 100 | 94 | 58 | 50 | 19 | 8 | 8 | 75 | 71 | 67 |
| 26 | heart failure – diagnostics | 32 | 28 | 88 | 96 | 96 | 54 | 74 | 33 | 11 | 33 | 67 | 70 | 63 |
| 27 | heart failure – weight measurement | 32 | 26 | 81 | 92 | 93 | 69 | 56 | 36 | 20 | 24 | 72 | 80 | 56 |
| 28 | atrial fibrillation – thyroid function | 32 | 25 | 78 | 98 | 99 | 64 | 54 | 50 | 25 | 29 | 75 | 83 | 71 |
| 29 | atrial fibrillation – oral anticoagulation | 32 | 27 | 84 | 99 | 99 | 59 | 31 | 42 | 19 | 15 | 73 | 77 | 69 |
| Neuropsychiatric diseases | ||||||||||||||
| 30 | ADHD – criteria for diagnosis | 51 | 26 | 51 | 96 | 96 | 35 | 65 | 42 | 27 | 27 | 54 | 62 | 50 |
| 31 | ADHD – educational support | 51 | 29 | 57 | 100 | 99 | 41 | 79 | 24 | 17 | 17 | 52 | 62 | 59 |
| 32 | ADHD – index prescription | 51 | 24 | 47 | 97 | 97 | 58 | 63 | 42 | 17 | 29 | 79 | 92 | 75 |
| 33 | ADHD – follow-up visits | 51 | 29 | 57 | 99 | 99 | 55 | 72 | 59 | 31 | 45 | 72 | 83 | 69 |
| 34 | depression – screening for Chronic Heart Disease and/or diabetes | 42 | 18 | 43 | 78 | 83 | 67 | 59 | 18 | 12 | 18 | 31 | 44 | 31 |
| 35 | depression – assessment of severity | 63 | 36 | 57 | 90 | 92 | 50 | 61 | 22 | 22 | 31 | 56 | 61 | 61 |
| 36 | dementia – screening for depression | 42 | 23 | 55 | 95 | 95 | 57 | 65 | 26 | 30 | 26 | 61 | 61 | 52 |
| 37 | dementia – medication review | 42 | 25 | 60 | 95 | 91 | 64 | 72 | 32 | 24 | 36 | 75 | 76 | 68 |
| 38 | dementia – available support | 42 | 26 | 62 | 96 | 91 | 62 | 65 | 23 | 15 | 19 | 58 | 65 | 50 |
| 39 | epilepsy – information on antiepileptics | 51 | 29 | 57 | 95 | 91 | 59 | 72 | 17 | 21 | 24 | 48 | 66 | 45 |
| 40 | epilepsy – recording seizure frequency | 51 | 27 | 53 | 97 | 98 | 63 | 70 | 22 | 19 | 26 | 63 | 78 | 63 |
| 41 | epilepsy –seizure prevention | 51 | 27 | 53 | 95 | 92 | 69 | 70 | 26 | 19 | 22 | 58 | 81 | 54 |
| Musculoskeletal diseases | ||||||||||||||
| 42 | low back pain – red flags | 42 | 34 | 81 | 99 | 96 | 74 | 71 | 35 | 18 | 29 | 67 | 74 | 62 |
| 43 | arthritis – analgesics | 42 | 35 | 83 | 97 | 94 | 68 | 66 | 20 | 6 | 18 | 32 | 38 | 29 |
| 44 | rheumatoid arthritis – disease modifying anti-rheumatic drug therapy | 42 | 30 | 71 | 97 | 97 | 80 | 73 | 53 | 37 | 47 | 60 | 83 | 63 |
| 45 | rheumatic arthritis – monitoring of side effects | 42 | 32 | 76 | 98 | 99 | 69 | 66 | 47 | 31 | 47 | 63 | 78 | 69 |
| 46 | rheumatic arthritis – treatment information | 42 | 23 | 55 | 100 | 98 | 61 | 65 | 43 | 24 | 23 | 70 | 61 | 57 |
| 47 | rheumatic arthritis – diagnostics | 42 | 19 | 45 | 95 | 95 | 74 | 63 | 42 | 17 | 21 | 50 | 68 | 47 |
| Otolaryngology | ||||||||||||||
| 48 | Presbyacusis hearing aid | 30 | 21 | 70 | 95 | 95 | 65 | 65 | 35 | 15 | 32 | 42 | 68 | 47 |
| mean percentage of all 48 QI mean values | 64 | 94 | 93 | 61 | 57 | 32 | 22 | 28 | 61 | 69 | 58 | |||
Applicability of indicators: almost two thirds (64%) on average of all practices across all QIs and specialist groups involved reported that the QIs presented to them apply to their medical practice, i.e. cover elements of practice activities relevant to patient care. The five QIs in the section practice management that were submitted to all specialist groups are rated as applying by almost 90%, with the exception of the QI “practice offers home visits”. In the sections “multidisciplinary topics”, “drug safety” and “immunisation”, contained in a variety of field-specific QI sets, the response to whether they apply to the practice fluctuates between 19% and 62%. Figures given for each QI in the five disease-specific sections fluctuate between 5% and 100%.
94% of medical practices reported for those QIs that apply to them that they actually collect the data required for the respective QI.
Availability of indicators: 93% on average of medical practices document QI information referring to applicable QIs. 61% have the required information available in electronic form.
Retrievability of data: the information required for denominator and numerator is retrievable from physicians in 57% of cases, and from the practice management system in 32% of cases. Depending on the QI there are notable differences in the retrievability of denominator-relevant information (0% to 79%) and of numerator-relevant information (0% to 59%).
Complexity of data retrieval: 22% of practices on average reported a period of up to 30 minutes as required to retrieve pertinent patient data from one year relevant to numerator and denominator of one QI. 78% of medical practices require more than 30 minutes or are unable to quantify the time required. 28% rate the time expenditure as reasonable, 72% as unreasonable.
Relevance of indicators: 61% of respondents assess the QIs as meaningful and therefore relevant to evaluate the quality of care of the QI-related area in their specialist field. This assessment value is far lower (30%) for some quality indicators such as “documented recommendation for smoking cessation”, “depression screening in patiens with coronary heart disease or diabetes” and “intake of freely available analgesics by patients with arthrosis”.
Reliability of indicators: 69% of medical practices rated the quality indicators as reliably surveyable.
Acceptance of indicators: 58% of practices on average accept the QI to assess the quality of care provided by a practice of their specialist field. Lower acceptance figures were found mainly for those quality indicators which also had a below-average rating for relevance.
Assessment of 9 field-specific QI sets
Tab. 3 lists the mean values for each indicator question per specialist field. The bottom line gives mean values across all specialist field mean values. For each specialist group the table indicates the average percentage of positive answers from all respondents according to question 1-10 across all quality indicators contained in the field-specific QI set.
Table 3Assessment results for QI sets according to specialist field.
| Specialist field number of applying QIs number of respondents per QI | applies | data collected | data documented | data computerized | denominator retrievable | numerator retrievable | <30 minutes required | reasonable amount of time | relevant | reliable | acceptable |
|---|---|---|---|---|---|---|---|---|---|---|---|
| gynaecology 17 QI n = 3-9 | 73% | 97% | 91% | 62% | 60% | 37% | 12% | 44% | 71% | 83% | 67% |
| otolaryngology 10 QI n = 3-10 | 68% | 95% | 80% | 40% | 51% | 37% | 13% | 31% | 72% | 85% | 81% |
| internal medicine 27 QI n = 2-12 | 58% | 93% | 93% | 36% | 51% | 36% | 10% | 28% | 79% | 79% | 71% |
| paediatrics 18 QI n = 3-11 | 77% | 97% | 97% | 39% | 66% | 34% | 13% | 36% | 76% | 74% | 71% |
| child and youth psychiatry 14 QI n = 1-10 | 62% | 99% | 99% | 27% | 82% | 54% | 5% | 24% | 86% | 87% | 87% |
| neurology and psychiatry 21 QI n = 1-10 | 59% | 99% | 98% | 47% | 67% | 45% | 23% | 31% | 66% | 77% | 74% |
| orthopaedics 13 QI n = 2-10 | 66% | 99% | 99% | 89% | 65% | 58% | 14% | 48% | 67% | 76% | 63% |
| urology 13 QI n = 1-10 | 68% | 75% | 73% | 45% | 21% | 18% | 15% | 27% | 63% | 70% | 65% |
| General practitioner 46 QI n = 2-20 | 72% | 93% | 91% | 74% | 59% | 27% | 19% | 26% | 52% | 63% | 47% |
| mean of specialist percentages | 67% | 94% | 91% | 51% | 58% | 38% | 16% | 33% | 70% | 77% | 70% |
Applicability of indicators contained in field-specific sets: an average of two thirds of all practices in one specialist field rate the indicators of their field-specific set as applying. Figures for internal medicine and neurology/psychiatry are almost 60%, whereby the group compositions with specific subgroups (cardiology, gastroenterology, rheumatology) with different specializations must be taken into account. Applicable indicators are then collected by 94% on average depending on group and indicator set, urology being the only group with a figure below 90% (75%).
Availability of indicators in the field-specific sets: in 91% on average of all field-specific sets the collected data are documented; half of the respondents report electronic documentation exclusively.
Retrievability of data: The information contained in the denominator and numerator of each indicator may be retrieved from the practice management system to an average of 58% and 38% respectively, depending on the specialist field. The only specialist group for which the figures notably deviate downwards is urology with 21% and 18%.
Complexity of data retrieval: 16% of indicators contained in one field-specific set are retrievable from the practice management system within less than 30 minutes. One third on average of respondents in all specialist fields rates the amount of time required as reasonable and acceptable.
Relevance of indicators in the field-specific sets: 70% of indicators in a field-specific set are rated as relevant to medical care provided in the respective area. Figures given by general practitioners are lowest with 52%.
Reliability of indicators in the field-specific sets: 77% of indicators contained in a field-specific set are rated as reliably surveyable; the rating given by general practitioners is the lowest with 63%.
Acceptance of indicators in the field-specific sets: Medical practices accept 70% of indicators contained in a field-specific set for an evaluation of quality of care. The figures given by general practitioners are again the lowest with 47%.
Discussion
Our study of the feasibility of 48 agreed quality indicators (QI) to evaluate the quality of outpatient care provided in the German health system reveals that there is still a long way to go until quality-based performance assessment will be possible. The results of our study show problems especially with the assignment of QI to specialist fields, the electronic availability and the time needed to retrieve the QI data.
The following section elucidates and critically examines individual findings on the feasibility criteria under exploration and their possible implications.
Applicability determines whether a medical practice actually has the patients or patient care structures corresponding to the quality indicator, if not, physicians were asked not to evaluate this QI. However, participants use this first criterion as a filter question and report whether they believe the respective quality indicator covers a care-related field of work in their medical practice. In this way physicians assess in fact the NASHIP expert group's allocation of quality indicators to field-specific sets. This assessment is particularly distinctive for quality indicators in the section “multidisciplinary topics” contained in nearly all field-specific QI sets. They address general (secondary) preventive issues. The two quality indicators on tobacco use and smoking cessation which apply to patients of practically all specialist fields are rated as applying to their own practice by only 60% and 40% respectively, the section “immunisation” shows comparable results, and in field-specific areas such as gynaecological quality indicators, the preventive QI “smoking cessation in pregnancy” is rated as applying even by only 27% of practices. Obviously the reason is not that a practice does not offer antenatal care but that the concept of accountability differs from the view of the expert panel. On the other hand, low ratings for applicability (5% - 13%) given by gynaecologists and general practitioners for the 3 HIV-related quality indicators may be due to the fact that the practice in question really has no patients with this diagnosis. A completely different picture emerges for quality indicators contained in only one or very few field-specific sets, such as “cervical screening follow-up” in the gyneacology group. A large majority of physicians rate them as applying being in accordance with the expert panel's consensus. The physicians’ broader interpretation of applicability may insofar be used to complement the decisions reached by the expert panel and to revise field-specific QI sets.
The second part of the feasibility assessment, enquiring whether data required for numerator and denominator of the QI is actually collected, receives a high level of positive responses with an average of 94% per QI and specialist field. This confirms that physicians rate those quality indicators which they deem applicable as relevant to patient care; in this regard the practitioners’ assessment is consistent with that of the expert panel.
Availability of data explores the extent and format of documenting the data required for indicator calculation. Findings show that 93% of medical practices document information regarding the applying quality indicators, so that a core criterion of feasibility is basically met. However, the format of availability is essential for the following criterion, retrievability. 98% report having an electronic medical record system, which is higher than the 72% found in the Commonwealth Fund survey in 2009 [
[19]
]. But an average of only 58% of medical practices have exclusively computerized documentation of QI-relevant data. These results reveal that it is essential for future implementation to encourage the widespread use of computerized documentation systems and thereby to improve the general availability of required data. But which data are in fact retrievable?Retrievability refers to the technical possibilities of practice management programmes, their user friendliness and also user competences. Denominator-specific data – mainly basic sociodemographic or diagnostic data - are retrievable by more than 50%. But only one third is able to retrieve the often more complex numerator-specific information. This result is similar to the one found by Marshall et al. [
[8]
] in their project “Measuring General Practice”. And it suggests that targeted data queries as an indispensable prerequisite for precisely defined quality indicators have not been necessary in primary care practices so far, and are not the norm e.g. in internal quality management procedures, not even in medical practices where 72% have a certified quality management system. To improve retrievability, it will be important to expand the technical possibilities of practice management systems, their user friendliness and also the competences of users.Respondents generally rate the complexity and expenditure of time required for indicator data collection as high. Only 20% across all specialist fields claim to need less than 30 minutes; hence it is easy to understand that two thirds of physicians surveyed state the effort involved to be unreasonable.
The last three feasibility criteria refer to the assessment of quality indicators from the practitioner's point of view. Relevance receives high ratings with 70% on average across all specialist fields, the same applies to reliability of data enquiry with 77%, and acceptance to be evaluated by these indicators with 70%. The downward deviation for general practitioners with regard to these quality indicators is notable; assessment values are lowest in this group. The question is whether it is possible to campaign among this group for more acceptance of quality indicators as tools to assess the quality of care provided since acceptance is a basic element of feasibility [
[29]
].The results of our study confirm the statement that the science and practice of quality indicator based quality measurement in ambulatory care in Germany shows much room for improvement [
14
, 15
]. Developing sets of quality indicators that are evidence-based, valid, reliable, tested and accepted in practice needs much more effort than has been spent so far. Even NICE has not been able to develop indicators in areas that need new data collection [[7]
]. So one of the biggest challenges will be to prioritize those areas of ambulatory care that are most important for driving quality improvement and make the data available that are needed to measure respective quality indicators.From a methodological point of view we regard our feasibility concept and questionnaire as widely employable. With its first criteria ‘applicability’ and ‘availability’ which evaluate if a specific quality indicator is at all part of the daily routine and being documented, it serves as a basic screening instrument to decide if implementing a specific indicator is actually possible in the respective institution. It is of high value especially when information on clinical practice, data documentation and infrastructure is not sufficiently available in a given health care context before implementation. Both, concept and questionnaire are not setting specific but can be employed in divers ambulatory as well as hospital contexts. What has to be considered is whom to survey. Usually it will be the clinically working physicians but it might be necessary to involve IT-specialists particularly in larger institutions where centralized queries are reasonable.
Limitations
The sample of 103 respondents deviates in several respects from a random sample taken from all German ambulatory care physicians. The majority of participants are male, work in joint or group practices, and – a most notable aspect – an exceptionally high percentage of 72% work with a certified quality management system. Implications for this study are that the convenience sample may bear more reference to the subjects under investigation, i.e. quality indicators and patient record management, than might be expected from a representative random sample. This may be reflected in familiarity with quality-related terminology (indicator, numerator, denominator etc.), in the type and extent of patient record documentation, in technical infrastructure, and in skills and competences which are required to retrieve documented information from the practice management system.
An interpretation of results might also need to take attitudes of professional organisations into account as a motivation to participate in the study since the study was known to have been commissioned by the NASHIP. This bias in the study means that a representative sample would in all likelihood produce even lower ratings for feasibility. The limitations to feasibility indicated in this study must be assumed to be underestimated.
Conclusions
The results of this feasibility study make further research advisable into the selection of quality indicators and collection of pertinent data, technical storage and retrievability, and into the acceptance of quality indicators by ambulatory care physicians. For this purpose it will be necessary to critically scrutinize the composition of field-specific indicator sets in order to improve the applicability of quality indicators as the basis of feasibility. The limited availability of required data indicates a need to encourage practice-based physicians to instal computerized management programs and use them for the entire documentation of patient records. The development, distribution and user-friendly integration of appropriate query engines in electronic data processing systems are indispensable prerequisites for implementation in Germany, in line with Marshall's conclusion [
[8]
]: “Only good information technology will enable the improvement agenda to move forward.” Another essential task is to foster among practitioners a general acceptance of QI-based practice assessments with regard to quality, range of services and reimbursement.Conflict of interest
No conflict of interest.
Appendix A. example of indicator questionnaire
Indicator Questionnaire
Indicator: Drug safety: long-term medication
- 1.Does this indicator apply to your practice?
- □yes
- □no
(If indicator does not apply to your practice, please do not fill out this questionnaire.) - □
- 2.Is information required for the numerator/denominator collected and documented?Tabled 1
Information data collected data documented yes no in part yes no in part number of long-term medication medication checked - 3.Is this information normally documented in writing or electronically?
- □written
- □computer based
- □in part written and in part computer based
- □
- 4.Can you access the information for the denominator from your PMS (practice management system) using a statistical search function?
- □yes
- □no
- □does not apply
- □
- 5.Can you access the information for the numerator from your PMS using a statistical search function?
- □yes
- □no
- □does not apply
- □
- 6.How long does it take to obtain information on all relevant patients in a single year from your patient files or electronic PMS?
- □_______ minutes
- □time required not possible to estimate
- □
- 7.Do you consider the amount of time required to obtain the relevant information to be reasonable under present practice conditions?
- □yes
- □no
- □do not know
- □
- 8.Do you consider the rating for quality of care achieved using the indicator to be of significance?
- □yes
- □no
- □do not know
- □
- 9.How do you rate the reliability in ascertaining this indicator?
- □low
- □medium
- □high
- □do not know
- □
- 10.Would you accept that an aspect of the quality of care within your specialist field would be rated using this indicator?
- □yes
- □no
- □do not know
- □
References
- Crossing the Quality Chasm: A New Health System for the 21st Century.The National Academies Press, 2001 ([http://www.nap.edu/openbook.php?record_id=10027])
- Methods for the guideline-based development of quality indicators--a systematic review.Implementation Science. 2012; 7: 21
- Research methods used in developing and applying quality indicators in primary care.BMJ. 2003; 326: 816-819
- Analyse und Evaluierung des QUALIFY Instruments zur Bewertung von Qualitätsindikatoren anhand eines strukturierten qualitativen Interviews. [Analysis and evaluation of the QUALIFY tool for assessing quality indicators with structured group interviews.].Z Evid Fortbild Qual Gesundhwes. 2011; 105: 38-43
- QUALIFY: Ein Instrument zur Bewertung von Qualitätsindikatoren. [QUALIFY - a Tool for Assessing Quality Indicators.].Z Arztl Fortbild Qualitatssich. 2008; 101: 683-688
- Choosing and evaluating clinical performance measures.Jt Comm J Qual Improv. 1998; 24: 470-479
- The NICE process for developing quality standards and indicators.Z Evid Fortbild Qual Gesundhwes. 2014; 108: 481-486
- Measuring general practice. A demonstration project to develop and test a set of primary care clinical quality indicators.The Nuffield Trust, London2003 ([http://www.nuffieldtrust.org.uk/publications/measuring-general-practice-demonstration-project-develop-and-test-set-primary-care-clin])
- Ambulatory care quality measures for the 6 aims from administrative data.Am J Med Qual. 2006; 21: 310-316
- The Comprehensive Care Project: Measuring Physician Performance in Ambulatory Practice.Health Services Research. 2010; 45: 1912-1933
- Anforderungsprofil an Qualitätsindikatoren. Relevanz aktueller Entwicklungen der Outcomes Forschung für das Qualitätsmanagement [Requirements for quality indicators. The relevance of current developments in outcomes research for quality management].Z Evid Fortbild Qual Gesundhwes. 2013; 107: 516-522
- Measuring quality of care: considering conceptual approaches to quality indicator development and evaluation.J Clin Epidemiol. 2013; 66: 1328-1337
- Quality indicators and performance measures: methods for development need more standardization.J Clin Epidemiol. 2013; 66: 1338-1339
- Aktuelle Herausforderungen der externen Qualitätssicherung im deutschen Gesundheitswesen. [Current challenges of external quality assurance in the German healthcare system].Z Evid Fortbild Qual Gesundhwes. 2013; 107: 523-527
- Sektorenübergreifende Qualitätssicherung in der ambulanten Versorgung. [Cross-sectoral quality assurance in ambulatory care].Z Evid Fortbild Qual Gesundhwes. 2013; 107: 528-533
- KBV entwickelt Starter-Set ambulanter Qualitätsindikatoren - AQUIK®-Set. [AQUIK®: Starter set of ambulatory quality indicators developed by the German National Association of Statutory Health Insurance Physicians.].Z Evid Fortbild Qual Gesundhwes. 2011; 105: 54-63
- NASHIP Develops a Starter Set of Ambulatory Quality Indicators Results of the "AQUIK®- Ambulatory Quality Indicators and Key Measures” Study.Kassenärztliche Bundesvereinigung, Berlin2009 ([http://www.kbv.de/aquik.html])
- A Survey Of Primary Care Doctors In Ten Countries Shows Progress In Use Of Health Information Technology.Less In Other Areas. Health Aff. 2012; 31: 2805-2816
- A Survey Of Primary Care Physicians In Eleven Countries, 2009: Perspectives On Care, Costs, And Experiences.Health Aff. 2009; 28: w1171-w1183
- On The Front Lines Of Care: Primary Care Doctors’ Office Systems, Experiences, And Views In Seven Countries.Health Aff. 2006; 25: w555-w571
- Primer on indicator development and application. Measuring quality in health care.JCAHO, Oakbrook Terrace1990
- A strategy for the implementation of a quality indicator system in German primary care.Qual Prim Care. 2011; 19: 183-191
- Validity versus feasibility for quality of care indicators: expert panel results from the MI-Plus study.Int J Qual Health Care. 2010; 22: 201-209
- Developing clinical and health improvement indicators for the Quality and Outcomes Framework (QOF) - Interim process guide.NICE, London2009 ([http://164.177.143.179/media/742/32/QOFProcessGuide.pdf])
- QUALIFY: Instrument for the Assessment of Quality Indicators Version 1. 0 (English).BQS, Düsseldorf2007 ([http://www.bqs-institut.de/images/stories/doc/106_QUALIFY-english-v10.pdf])
- Selecting common measures of quality and system performance.Med Care. 2003; 41: I39-I47
- A Road Map For Improving The Performance Of Performance Measures.Health Affairs. 2011; 30: 569-573
- Qualitätsmanagement in kinder- und jugendärztlichen Praxen. [Implementation status of quality management systems in paediatric practices in Germany].Monatsschrift Kinderheilkunde. 2010; 159: 145-151
- Die Darstellung der hausärztlichen Versorgungsqualität durch Qualitätsindikatoren. [Using quality indicators to measure the quality of general practice care in Germany.].Z Evid Fortbild Qual Gesundhwes. 2011; 105: 13-20
Article info
Publication history
Accepted:
February 27,
2015
Received in revised form:
February 26,
2015
Received:
September 29,
2014
Identification
Copyright
© 2015 Published by Elsevier Inc.
User license
Creative Commons Attribution – NonCommercial – NoDerivs (CC BY-NC-ND 4.0) | How you can reuse 
Elsevier's open access license policy
Creative Commons Attribution – NonCommercial – NoDerivs (CC BY-NC-ND 4.0)
Permitted
For non-commercial purposes:
- Read, print & download
- Redistribute or republish the final article
- Text & data mine
- Translate the article (private use only, not for distribution)
- Reuse portions or extracts from the article in other works
Not Permitted
- Sell or re-use for commercial purposes
- Distribute translations or adaptations of the article
Elsevier's open access license policy
