Development of respiratory disease quality indicators for pediatric patients in Jordanian emergency departments: a RAND/UCLA appropriateness method

Introduction

The emergency department (ED) serves as a cornerstone unit within hospitals, playing a critical role in the healthcare system by providing care for patients with acute illnesses and critical injuries (1). However, the increasing dependency on EDs introduces numerous challenges that can adversely affect the delivery of high-quality care. These challenges include overcrowding, resource limitations, heightened workloads, and performance pressures (2-4). In Jordan, similar trends are underscored (5). Acute respiratory diseases account for a substantial number of hospitalizations and ED visits (6,7), making them among the most common and potentially severe conditions that lead pediatric patients to seek emergency care (8), including asthma, bronchiolitis, and croup (9). Previous studies indicate a lack of standardized measures for assessing the quality of hospital care provided for respiratory conditions (7). Furthermore, prior research has highlighted considerable variability in the clinical management of pediatric patients with respiratory diseases in EDs (10), which directly impacts the quality of care provided to these patients.

Quality in healthcare is described as the extent to which health services for individuals and populations enhance the probability of achieving desired health outcomes and align with current professional knowledge (11). In this milieu, quality indicators (QIs) are vital elements, serving as benchmarks for evaluating care quality and enabling healthcare providers and researchers to identify areas requiring improvement and inform policy development (12,13). The Agency for Healthcare Research and Quality (AHRQ) defines QIs as “standardized, evidence-based measures of healthcare quality that can be used with readily available hospital inpatient administrative data to measure and track clinical performance and outcomes” (14).

The imperative to enhance healthcare quality through effective performance measurement is critical for managing healthcare systems globally (15). Without robust and precise measurement, it is impossible to objectively assess the quality of healthcare services with methodological rigor and validity or to evaluate whether quality improvement initiatives have their desired effect or if there are adverse outcomes from making modifications (16). Research supports the feasibility of transferring QIs between countries, with varying levels of agreement on their adaptation for local use (17). For instance, experts in England found 79 out of 93 selected The Assessing Care of Vulnerable Elderly (ACOVE) indicators to be suitable for local use without any change (18). However, research has underscored the importance of modifying existing QIs or developing new ones to accurately reflect the unique cultural, demographic, and geographic characteristics of each local healthcare setting (17). These indicators should meet essential criteria, including feasibility, relevance, and scientific validity, preferably supported by rigorous studies (19,20).

The adaptation and local development of QIs for respiratory diseases in Jordanian EDs are crucial to enhancing care processes and outcomes. This study is part of a broader research project focused on developing and testing QIs for pediatric care related to three respiratory diseases in Jordan. This approach is particularly vital in pediatric emergency care, where local factors significantly influence the quality of care and patient outcomes. Therefore, this study aims to use an evidence-informed expert panel consensus process to identify valid and feasible QIs for pediatric asthma, bronchiolitis, and croup in Jordanian EDs. We present this article in accordance with the STROBE reporting checklist (available at https://jeccm.amegroups.com/article/view/10.21037/jeccm-24-90/rc).

Methods

Study design and sitting

This study used the Research and Development/University of California, Los Angeles (RAND/UCLA) appropriateness method (RAM) (21), using two rounds of anonymous questionnaires to establish consensus on the final selection of indicators. This method combines scientific research with expert opinions and has become a widely recognized approach for reaching a formal agreement on the best healthcare practices. The study was carried out in several healthcare sectors in Jordan, chosen to represent a variety of urban, suburban, and rural healthcare environments.

Selection of participants (panelists) and setting

In line with RAM (21), an expert panel of 14 members, including general practitioners and consultants, actively participated in this study, as detailed in Table 1. The experts were recruited from the Ministry of Health (MOH), Jordan University Hospital (JUH), and King Abdullah University Hospital (KAUH). The main selection criteria for the panelists included acknowledged leadership in pediatric care, experience in the “quality” field, possession of a Jordanian Board Certificate, absence of conflicts of interest, and geographic diversity. Specifically, efforts were made to include representatives from both public and university hospitals.

Preliminary work

Preparation of QIs

A previous systematic review and meta-analysis were conducted to identify existing QIs specifically designed for use in EDs for pediatric respiratory conditions with asthma, bronchiolitis, and croup. We performed searches in four electronic databases: Web of Science, Scopus, CINAHL, and MEDLINE. We used Medical Subject Heading (MeSH) terms and keywords related to our topic, including “quality indicator”, “emergency department”, “respiratory illnesses”, and “pediatric patient” (22,23). In total, the review identified 255 QIs extracted from expert panel studies, observational studies, and randomized clinical trials (RCTs) addressing both clinical and non-clinical aspects of pediatric respiratory care in emergency services.

A total of 124 indicators, derived from expert panel studies that combined expert opinions, literature reviews, and field testing, were presented to our panel. Furthermore, these QIs were categorized according to two frameworks: Institute of Medicine (IOM) (24), which encompasses six domains of healthcare quality: effectiveness, timeliness, efficiency, safety, equity, and patient-centeredness, and Donabedian’s including structure, process, and outcome (25). Table 2 details how the identified attributes were aggregated into these domains and their healthcare relationship.

Data collection (consensus process)

In the first round, conducted in February 2024, panelists received all QIs for three respiratory conditions. They were asked to use their experience and knowledge to rate each QIs for validity and feasibility on a 9-point Likert scale (1 = low, 9 = high) (26), assessing their transferability to Jordanian EDs. ‘Validity’ refers to the presence of adequate scientific backing or professional agreement supporting the QIs as reliable metrics for assessing care quality. When an indicator receives a rating between 1 and 3, it indicates that it is not suitable for quality evaluation. Ratings from 4 to 6 suggest that the indicator’s ability to evaluate quality is uncertain. However, ratings between 7 and 9 confirm that the indicator is a valid criterion for assessing quality. Feasibility implies that the information required for these QIs is expected to be available in a standard ED medical record, whether paper-based or electronic, for assessing eligibility and compliance with the indicators. A rating between 1 and 3 indicates that using data from a typical medical record to evaluate the indicator is not feasible. Ratings from 4 to 6 suggest that there may be significant variation in the accessibility of the necessary information to assess the indicator. Conversely, ratings from 7 to 9 indicate that it is feasible to consistently obtain the necessary information from the medical record to evaluate the indicator [supplement file for the first-round rating form (Appendix 1)].

Before the first round, panelists were briefed via email and phone calls about the study’s goals and were invited to contribute comments to improve QI clarity and propose any additional QIs. A pre-established decision rule was applied: any indicators rated ≤3 on both criteria in the first round by all panelists would be excluded from further consideration. The second round, conducted in March 2024, involved experts reassessing their ratings and addressing any discrepancies from the first round’s results. In this round, panelists had the opportunity to review each indicator’s ratings, including the median and mean absolute deviation (MAD) around the median, which were presented to show the distribution of all panelists’ initial ratings from the first round. They were then asked to re-rate each indicator using the same criteria. QIs that achieved consensus in the second round, indicated by a mean rating of ≥7 for validity, and ≥4 for feasibility, were included in the final list of QIs.

Statistical analysis

Data were analyzed using Microsoft Excel 2019 (Microsoft, Richmond, Washington, USA). The criteria for determining the validity and feasibility of the QIs were based on mathematical rules classically applied in the RAM (27). QIs were deemed valid if the median score was 7 or higher and feasible if the median score was 4 or higher (26), both conditions requiring the absence of significant disagreement. Conversely, a median score ranging from 1 to 3, without disagreement, indicated that the QIs were neither valid nor feasible. Disagreement was defined as ≥5 panelists scoring in each extreme section of the 9-point scale (1–3 and 7–9). Additionally, the MAD around the median was used as a continuous measure to analyze the dispersion and variability of the panelists’ ratings more effectively.

Ethical considerations

This study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). Ethical approval for this study was secured from the Institutional Review Board (IRB) at the KAUH (No. 2023/163/56) and MOH in Jordan (No. 2023/334). Informed consent is not required as no patients were involved in this study.

Results

A total of 124 QIs were evaluated by the panel for validity and feasibility, as illustrated in Figure 1, which details the selection process at various stages. In the first round, the panel consolidated five indicators into two and modified three others for better clarity and appropriateness. Additionally, 43 QIs were excluded in the first round based on a consensus about their validity and feasibility. In the second round, another 27 QIs were removed, resulting in a final set of 51 validated indicators, with 25 deemed not feasible, as shown in Table 3. This table also presents key statistics for each QI, including the median, the median absolute deviation (MAD) around the median (a measure of variability), the level of consensus, and the outcomes of the consensus process. Notably, 87% of the final indicators are process-oriented, while 13% are outcome-oriented, with no indicators categorized under structural elements. Most of these indicators focus on effectiveness and safety within the IOM domain. Table 4 summarizes the final 26 indicators that were validated as both valid and feasible.

Figure 1 Process of selecting indicators at different stages of the study. QI, quality indicator.

Discussion

While leveraging work from other settings offers significant advantages, QIs should not be transferred directly across locations without an intermediary process that facilitates any necessary adjustments to account for differences in clinical practices and professional culture (28). Furthermore, the process of developing QIs should systematically integrate scientific evidence to ensure their relevance and efficacy in the specific healthcare context (29). This study was designed to develop QIs specifically for pediatric patients with respiratory diseases in Jordanian EDs, utilizing a RAM (27). It has been successfully applied in previous healthcare research (20).

Out of 124 assessed QIs, our panel found 26 (or 21%) to be valid and feasible. These QIs are not only relevant but also actionable in Jordanian ED, offering a framework for improving pediatric respiratory care quality. The successful adaptation of QIs from international settings emphasizes the significance of contextualizing healthcare QIs (17,20). While international studies offer a robust foundation, it is imperative to adjust these indicators to reflect local clinical practices, cultural factors, and resource availability. For instance, the selected indicators for asthma, bronchiolitis, and croup, such as the percentage of bronchiolitis patients discharged home from the ED with a prescription for antibiotics, are directly vailed and feasible to the Jordanian context, ensuring that they address the most pressing needs within this specific healthcare setting. However, implementing these QIs in Jordanian EDs presents several challenges. The healthcare system in Jordan faces significant hurdles, including escalating demands for services, rising healthcare expenditures, a critical shortage of staff, inconsistent diagnoses, limited resources, and hospital overcrowding (5,30-32). All of which might impede the successful implementation of key QIs. Furthermore, the variability in clinical practices for managing pediatric respiratory diseases further complicates efforts to standardize care through QIs (10,33).

Within the Donabedian framework (25), the majority of these QIs were process-oriented (n=23; 88.5%), with a smaller number being outcome-oriented (n=3; 11.5%) Moreover, none of the QIs pertained to structural elements. The predominance of process-oriented indicators shows that these measures are more instantly actionable within the Jordanian healthcare system, paving the door for improved care delivery systems. Process indicators are critical for measuring the quality of care, as they are directly linked to the actions of healthcare providers, which can be changed to enhance patient outcomes (34). The assessment of professional practice quality primarily focuses on measuring the processes used in healthcare delivery or the outcomes achieved from the activities of healthcare professionals (35). However, the absence of structural indicators indicates a substantial weakness in assessing overall care quality. Structural indicators reveal a crucial gap in appraising the whole healthcare environment. These indicators, which evaluate the infrastructure, resources, and organizational components of treatment, are critical for attaining long-term patient outcomes. In contrast, within the IOM area (24), the primary emphasis was on effectiveness and safety indicators.

Moreover, this study discovered 25 indicators as valid but not feasible due to significant challenges in data availability within Jordanian EDs. These challenges primarily stem from inadequate documentation practices, where medical records do not consistently record essential data. According to our definition of feasibility, this lack of reliable data highlights a critical barrier to the effective implementation of QIs, as accurate and comprehensive data collection is essential for monitoring and improving the quality of care. To address these difficulties, Jordanian healthcare providers must urgently improve their data documentation and management methods.

The study represents a critical step and significant achievement in developing valid and feasible QIs tailored to the specific needs of pediatric patients with respiratory diseases in Jordanian EDs. However, this study did not include an external validation phase to assess the effectiveness of the identified QIs in real-world settings. This step is essential to confirm the acceptability, reliability and applicability of the QIs beyond the expert panel’s consensus (20). Future research should prioritize testing these QIs in Jordanian EDs and focus on developing and integrating more outcome and structural indicators to provide a more comprehensive assessment. Additionally, improving data management systems will enhance the feasibility of QIs in Jordan. By addressing these areas, we can improve the effectiveness of QIs in Jordan and make them more adaptable for broader use in regions with similar healthcare challenges.

There are a few limitations to our study. First, the expert panel consisted of only 14 members, primarily from Middle and Northern Jordan and the MOH, with limited representation from Southern Jordan and other institutions. This composition may have introduced biases, as the perspectives of healthcare providers from different regions or institutional backgrounds were not fully captured. The size and specific composition of the expert panel may not fully reflect the diversity of perspectives in pediatric respiratory emergency care, potentially introducing bias into the consensus-building process. While the consensus process was robust, it may have been influenced by the varied backgrounds and experiences of the panel members, potentially affecting the authenticity of their evaluations. Additionally, the QIs were specifically designed for pediatric emergency care in Jordan, which may limit their generalizability to other healthcare settings with different cultural, demographic, or health care systemic characteristics.

Conclusions

This study formulated 26 valid and feasible QIs for managing pediatric asthma, bronchiolitis, and croup in Jordanian ED. It underscores the essential role of expert consensus in shaping QIs and adapting them to the specificities of local healthcare environments. However, before implementation, these candidate QIs need to be tested.

Acknowledgments

The authors would like to thank all panelists for their time and expertise during the consensus process: Prof. Manar al-Lawama; associate Prof. Enas Al Zayadneh (JUH); associate Prof. Saif Aldin Rawabdeh (KAUH); associate Prof. Jomana Al Sulaiman; Dr. Mohammad Wahsheh; Dr. Aseel Bataineh; Dr. Haitham Oweis; Dr. Medhat Al-Bqai (Princess Rahma Hospital/MOH); Dr. Salah Qazaqi (Al Bashir Hospital/MOH); Dr. Laith Kamal Kilani (Princess Basma Hospital/MOH); Dr. Duaa Samara; Dr Mahmood Massad; and Dr. Emad Al-Hamad (Ramtha Hospital/MOH).

Funding: None.

Footnote

Reporting Checklist: The authors have completed the STROBE reporting checklist. Available at https://jeccm.amegroups.com/article/view/10.21037/jeccm-24-90/rc

Data Sharing Statement: Available at https://jeccm.amegroups.com/article/view/10.21037/jeccm-24-90/dss

Peer Review File: Available at https://jeccm.amegroups.com/article/view/10.21037/jeccm-24-90/prf

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://jeccm.amegroups.com/article/view/10.21037/jeccm-24-90/coif). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. This study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). Ethical approval for this study was secured from the Institutional Review Board (IRB) at the KAUH (No. 2023/163/56) and MOH in Jordan (No. 2023/334). Informed consent is not required as no patients were involved in this study.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. He J, Hou XY, Toloo S, et al. Demand for hospital emergency departments: a conceptual understanding. World J Emerg Med 2011;2:253-61. [Crossref] [PubMed]
2. Hansen K, Boyle A, Holroyd B, et al. Updated framework on quality and safety in emergency medicine. Emerg Med J 2020;37:437-42. [Crossref] [PubMed]
3. Darraj A, Hudays A, Hazazi A, et al. The Association between Emergency Department Overcrowding and Delay in Treatment: A Systematic Review. Healthcare (Basel) 2023;11:385. [Crossref] [PubMed]
4. Suleiman K, Hijazi Z, Al Kalaldeh M, et al. Quality of nursing work life and related factors among emergency nurses in Jordan. J Occup Health 2019;61:398-406. [Crossref] [PubMed]
5. Hammad EA, Alabbadi I, Taissir F, et al. Hospital unit costs in Jordan: insights from a country facing competing health demands and striving for universal health coverage. Health Econ Rev 2022;12:11. [Crossref] [PubMed]
6. Kassam S, Serrano-Lomelin J, Hicks A, et al. Geography as a Determinant of Health: Health Services Utilization of Pediatric Respiratory Illness in a Canadian Province. Int J Environ Res Public Health 2021;18:8347. [Crossref] [PubMed]
7. Reyes MA, Etinger V, Hronek C, et al. Pediatric Respiratory Illnesses: An Update on Achievable Benchmarks of Care. Pediatrics 2023;152:e2022058389. [Crossref] [PubMed]
8. Cavallaro SC, Michelson KA, D'Ambrosi G, et al. Critical Revisits Among Children After Emergency Department Discharge. Ann Emerg Med 2023;82:575-82. [Crossref] [PubMed]
9. Choi J, Lee GL. Common pediatric respiratory emergencies. Emerg Med Clin North Am 2012;30:529-63. x. [Crossref] [PubMed]
10. Knapp JF, Simon SD, Sharma V. Quality of care for common pediatric respiratory illnesses in United States emergency departments: analysis of 2005 National Hospital Ambulatory Medical Care Survey Data. Pediatrics 2008;122:1165-70. [Crossref] [PubMed]
11. Lohr KN, Schroeder SA. A strategy for quality assurance in Medicare. N Engl J Med 1990;322:707-12. [Crossref] [PubMed]
12. Aaronson EL, Marsh RH, Guha M, et al. Emergency department quality and safety indicators in resource-limited settings: an environmental survey. Int J Emerg Med 2015;8:39. [Crossref] [PubMed]
13. Guttmann A, Razzaq A, Lindsay P, et al. Development of measures of the quality of emergency department care for children using a structured panel process. Pediatrics 2006;118:114-23. [Crossref] [PubMed]
14. Agency for Healthcare Research and Quality (AHRQ). AHRQ QI Technical Documentation, Version v2024. Rockville, MD, USA. 2024.
15. Mannion R, Goddard M. Performance measurement and improvement in health care. Appl Health Econ Health Policy 2002;1:13-23. [PubMed]
16. Mainz J. Quality indicators: essential for quality improvement. International Journal for Quality in Health Care 2004;16:i1-2. [Crossref]
17. Zhu S, Wu T, Leese J, et al. What is the value and impact of the adaptation process on quality indicators for local use? A scoping review. PLoS One 2022;17:e0278379. [Crossref] [PubMed]
18. Steel N, Melzer D, Shekelle PG, et al. Developing quality indicators for older adults: transfer from the USA to the UK is feasible. Qual Saf Health Care 2004;13:260-4. [Crossref] [PubMed]
19. Campbell SM, Braspenning J, Hutchinson A, et al. Research methods used in developing and applying quality indicators in primary care. Qual Saf Health Care 2002;11:358-64. [Crossref] [PubMed]
20. Pap R, Lockwood C, Stephenson M, et al. Development of prehospital care quality indicators for the Australian setting: a modified RAND/UCLA appropriateness method. Emerg Med J 2022;39:57-62. [Crossref] [PubMed]
21. Fitch K, Bernstein SJ, Aguilar MD, et al. The RAND/UCLA Appropriateness Method User's Manual. Santa Monica, CA: RAND Corporation; 2000.
22. Alkhazali IE, Alrawashdeh A, Hashairi Fauzi M, et al. Quality Indicators of Pediatric Asthma Care in the Emergency Department; a Systematic Review and Meta-Analysis. Arch Acad Emerg Med 2024;12:e26. [PubMed]
23. Alkhazali IE, Alrawashdeh A, Fauzi MH, et al. Quality Indicators for Pediatric Bronchiolitis and Croup Care in the Emergency Department; a Systematic Review and Meta-Analysis. Archives of Academic Emergency Medicine 2024;12:e52. [PubMed]
24. Institute of Medicine Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington (DC): National Academies Press (US); 2001.
25. Donabedian A. The quality of care. How can it be assessed? JAMA 1988;260:1743-8. [Crossref] [PubMed]
26. Mangione-Smith R, Roth CP, Britto MT, et al. Development and Testing of the Pediatric Respiratory Illness Measurement System (PRIMES) Quality Indicators. Hosp Pediatr 2017;7:125-33. [Crossref] [PubMed]
27. Fitch K, Bernstein SJ, Aguilar MD, et al. The RAND/UCLA Appropriateness Method User's Manual. Santa Monica, CA: RAND Corporation; 2001.
28. Marshall MN, Shekelle PG, McGlynn EA, et al. Can health care quality indicators be transferred between countries? Qual Saf Health Care 2003;12:8-12. [Crossref] [PubMed]
29. Mainz J. Developing evidence-based clinical indicators: a state of the art methods primer. Int J Qual Health Care 2003;15:i5-11. [Crossref] [PubMed]
30. Alshyyab MA, Albsoul R, Fitzgerald G. Factors influencing patient safety culture in operating room in a teaching hospital in Jordan: a qualitative descriptive study. The TQM Journal 2023;35:1722-42. [Crossref]
31. Al Hijaa MRA. Strategic Management’s Influence on Hospital Performance: A Comprehensive Study of Jordanian Healthcare Context. European Journal of Business and Management Research 2023;8:114-9. [Crossref]
32. Amar M. Quality of health services in Jordanian government hospitals Reality and challenges. Journal of University Studies for inclusive Research 2018;2:20-36.
33. Campbell SM, Roland MO, Buetow SA. Defining quality of care. Soc Sci Med 2000;51:1611-25. [Crossref] [PubMed]
34. Mainz J. Defining and classifying clinical indicators for quality improvement. Int J Qual Health Care 2003;15:523-30. [Crossref] [PubMed]
35. Rubin HR, Pronovost P, Diette GB. The advantages and disadvantages of process‐based measures of health care quality. International Journal for Quality in Health Care 2001;13:469-74. [Crossref] [PubMed]