Evaluation of outcome and performance of an intensive care unit in Hong Kong by APACHE IV model: 2007–2014

Introduction

Intensive care unit (ICU) service is both labour and resources intensive as it manages patients who are critically ill. It was estimated that the cost of care of a patient in ICU was 6 times as much as in general ward (1). As a result, it is important to monitor the quality of care and performance of ICU service.

Various overseas organizations were established to audit the performance of ICUs in their own countries (2-5). They define a number of indicators which are used for monitoring and benchmarking the outcomes and quality of care (6-8). Such indicators include occupancy rate, crude hospital mortality rate, risk adjusted hospital mortality rate, length of stay (LOS) (9-13), admission rate of low risk monitoring and readmission rate (14,15).

While crude mortality data may offer some guidance to ICU performance, risk-adjusted mortality according to disease severity can improve the assessment (16-18). One of the common tools is Acute Physiology and Chronic Health Evaluation (APACHE) model (19,20). It provides a scoring system to define the severity of illness taking into account of the degree of physiological derangement as well as the chronic health status of the patients. Accordingly, it estimates the risk of death after adjustment for the severity of illness. Such severity-adjusted indicators can be used to assess performance of a single ICU over time or to compare individual units.

In Hong Kong, Hospital Authority (HA) adopted the APACHE IV model for monitoring the outcome and performance of ICUs. The electronic version of the APACHE form was developed in 2006, enabling automatic capture of physiological data.

Such clinical registry collects a standardised set of de-identified data from contributing ICUs. Since data management is independent of the providers, it enables accurate interpretation of results. The database contained useful information on the availability and utilisation of ICU resources, patient outcomes, disease patterns and the effectiveness of ICU interventions. As a result, it serves as a valuable tool for quality assurance. Moreover, it facilitates development and implementation of strategies to improve intensive care services delivery in Hong Kong.

Methods

Data analysis

Our hospital is a tertiary referral centre with more than 1,500 beds. It is a regional centre for trauma, major surgeries and many medical subspecialties in Hong Kong. Its ICU provides both medical and surgical intensive care services. The aim of this study was to benchmark the outcome and performance of our ICU based on the APACHE IV model.

All first admissions to our ICU from 2007 to 2014 were collected. Patients with unknown hospital vital outcome and data of discharge, an ICU LOS of fewer than 4 hours, and patients less than 16 years of age were excluded for analysis. The worst physiological variables within the first 24 hours after ICU admission were collected. These data were recorded by clinical information system (CIS) of ICU and were automatically uploaded to the database.

The expected LOS and mortality rate were derived from APACHE acute physiological score (APS), chronic health, type of admission, lead time by the APACHE IV model and disease categories. The LOS ratio was calculated as the ratio of actual LOS to the expected LOS and the standardized mortality ratio (SMR) as the ratio of actual mortality to expected risk of death. Patients discharged to other ICUs, with non-scoring APACHE diagnosis, including burns, CABG and medical diagnosis not specified were excluded for calculation.

Statistical methods

Data were analysed using the Statistical Package for the Social Sciences (Windows version 15.0; SPSS Inc., Chicago, IL, US). Descriptive statistics were used to report the results. Variables were reported using mean and standard deviation (SD) (21), except where otherwise indicated. Chi square test was used for comparisons of categorical data respectively. All P values of less than 0.05 were considered significant.

Results

Demographics

The database, exclusive of ICU readmissions for both the same and separate hospital admission (n=383), age less than 16 or missing (n=2) and patients having ICU LOS less than 4 hours (n=99), contained records of 8,037 patients for analysis. A total of 7,885 patients were included for calculation of APACHE IV SMR and LOS ratio as 31 patients were discharged to other ICUs and 121 patients had non-scoring APACHE IV diagnosis.

The annual ICU admissions were about 1,000. The mean age of patients admitted to our ICUs ranged from 57.87 to 59.98 years old and there was male predominance (Table 1). The distribution of age showed a bimodal pattern, which peaked in the range of 46–55 years and 76–85 years (Figure 1). Most of the patients admitted to ICU did not have chronic health problems as defined by the APACHE IV model. From the study, approximately half of the admissions were from operation theatres and the other half from general wards, emergency department or other hospitals. Among the post-operative cases, almost half of them underwent elective surgeries and the other half had emergency surgeries. The proportion of admission from emergency department was quite constant at around 10–15% except in 2007 (Table 1).

Table 1 Demographics and outcome of patients admitted to ICU 2007–2014
Full table

Figure 1 Distribution of age, APACHE IV APS and length of stay of patient in ICU and their correlations with crude hospital mortality rate: 2007–2014. APACHE, Acute Physiology and Chronic Health Evaluation; APS, acute physiological score; ICU, intensive care unit.

From 2007 to 2014, the mean APACHE IV APS ranged from 57.78 to 64.01. The distribution of APS skewed to the left with the peak in the range of 21–30. The incidence then declined with increase in the APS but it rose slightly again in the range greater than 120 (Figure 1)

Outcome and performance

The total readmission rate was in the range of 4–5%. Among them, about 1.5% of the patients were readmitted within 72 hours. There was a discrepancy between median LOS and mean LOS. The median LOS was around 2 days while the mean LOS was in the range of 4–5 days. The majority of patients stayed in ICU in the range of 0–2 days and only a small proportion of patients stayed in ICU over 14 days (Figure 1). The overall mean LOS ratio was about 1 from 2007 to 2014 (Table 2).

Table 2 Risk-adjusted outcomes in terms of LOS ratio and SMR: 2007–2014
Full table

The crude ICU mortality rate was about 10% and the crude hospital mortality rate was around 20% (Table 1). The crude hospital mortality rate of elective postoperative patients was much lower than that of emergency postoperative and non-operative patients (Table 3). From 2007 to 2014, the crude hospital mortality rate for elective postoperative cases was 4.01%, compared to 19.82% for emergency post-operative cases (OR =4.95, 95% CI 3.85–6.36, P<0.001) and 23.50% for non-operative cases (OR =5.87, 95% CI 4.62–7.44, P<0.001). In addition, the crude hospital mortality rate was found to have a positive association with age, APACHE APS and LOS (Figure 1). The overall SMR was about 0.8 during the study period.

Table 3 Crude hospital mortality rate of elective postoperative, emergency postoperative and non-operative patients: 2007–2014
Full table

From 2007 to 2014, the outcome in term of readmission rate, LOS, ICU mortality and crude hospital mortality were quite constant. Moreover, there was no wide fluctuation in the outcome of patients in term of SMR and LOS ratio after adjustment of severity of illness according to the APACHE IV model.

Discussion

Variation in patient characteristics, including admission source, parent specialties, APS and disease categories may affect the outcome and performance of ICUs, such as average LOS and crude hospital mortality rate (22-24). Such differences may be related to the referral pattern of the parent specialties and admission policy of ICUs, which in turn affect the access of critical patients to ICU. It is partly reflected by the APS of the patients which is an indicator of the severity of the illness (25,26). For example, for patients with severe sepsis, the outcome can be improved by early resuscitation and early antibiotics administration. It was advocated that such aggressive treatment should be started early at emergency department and the patient should be directly transferred from emergency department to ICU as soon as possible. In this way, it may lead to less physiological derangement and may have a positive impact on the outcome of the patients (27).

High readmission rate of ICU may be due to premature discharge as a result of high occupancy rate (28). It was found that a high ICU occupancy rate had an adverse effect on the mortality rate (14,18,29). A high occupancy rate may lead to forced early discharge, high readmission rate or denial of admission (30) for the critically ill patients. In some studies, the recommended occupancy rate was about 80% (28). The ideal occupancy rate should not be exceeded to ensure no denial of or delay in ICU admission (10). Critically ill patients should gain access to ICU whenever it was indicated according to objective assessment of physiological criteria. On the other hand, low occupancy rate may reflect underutilization of ICU resources.

There is no single appropriate statistical indicator for LOS so both mean and median LOS were provided in this report. The distribution of LOS is usually skewed to the left as a higher proportion of patients are discharged early. The median LOS is a better reflection of the distribution as it reflects the LOS of the majority of the patients. The mean LOS is a more superior indicator of bed demand and resources (22,28). If an ICU admits a higher proportion of patients with lower APACHE score, there may be a greater discrepancy between the median and mean LOS as the LOS of such patients is very short, usually in terms of 1 to 2 days.

The LOS is affected by such factors as disease categories and severity, etc. Therefore, risk-adjusted LOS ratio is a better reflection than the actual LOS. When an individual ICU has a satisfactory SMR, LOS of ICU survivors lower than expected might result from effective and efficient provision of care or early discharge of patients to facilities of intermediate care (15). Conversely, in the presence of a satisfactory SMR, difference in LOS of non-survivors may arise from discrepancies in limitation of life sustaining treatment in case of medical futility (28).

Understandably, age, APACHE IV score and LOS correlated with mortality and it was consistent with the findings of other studies (5,11,13,14). In general, those ICUs with a lower APACHE score tend to have a lower ICU mortality rate. If an ICU admits more elective post-operative patients, the crude mortality rate is lower as their risk of death of such patients is low (31). Our ICU is a tertiary referral hospital and many ultra-major operations are performed. These patients are admitted to the ICUs for close monitoring post-operatively as they have a higher chance of developing major surgical complications. They are usually stable after operation so their APACHE IV APS is lower. Development of major complications is uncommon so their mortality rate is much lower compared to emergency operative or non-operative cases.

Support from other specialties may also play a crucial role (15). For operative patients, earlier detection of surgical problems and earlier interventions before major deterioration will lead to improvement of outcome. Therefore, adequate support from other specialties may also affect the outcome of ICU patients.

Many studies showed that there was loss of calibration of APACHE model over time with a trend of gradual reduction in SMR (10,25). In a study in Australia and New Zealand, it was found that there was steady decrease in SMR over 10-year period (16). It was suggested to be related to changes in actual mortality rate and case-mix of patients. Other possible contributing factors included invention of new treatment modalities and adoption of new treatment guidelines. However, such trend of reduction in SMR was not evident in our study.

The strength of our study is high accuracy of the data as most of the data were captured by computer automatically. It significantly eliminated human errors of data transcription. However, there are certain limitations of current study. One of such limitations is that the APACHE model was developed in USA and its application in ICUs outside USA has not been validated (10,11,13). It is doubtful whether such model can be applied to our patients in Hong Kong and calibration of the model with our local patients may be needed. In a local study, it was found that the APACHE II and SAPS II models provided good discrimination power but the calibration was poor for the ICU patients in Hong Kong (32).

Conclusions

This study employed validated risk-adjusted APACHE IV model to evaluate the outcome and performance of an ICU in a tertiary centre in Hong Kong. It was found that the case-mix of our ICU was quite constant during the study period. About half were medical cases and half were surgical cases. For the surgical cases, around 50% underwent elective surgeries and the others had emergency surgeries. Almost 10–15% of admissions were directly from emergency department. There was no marked fluctuation in the severity of illness of our patients. The average LOS was about 5 days and the crude hospital mortality rate was about 20%. After risk adjustment, the SMR was about 0.8 and the LOS ratio was about 1.

This study showed that APACHE IV model was useful in longitudinal monitoring the outcome and performance of our ICU and there was no wide fluctuation of outcome in terms of SMR and LOS ratio from 2007 to 2014.

Acknowledgements

The authors thank Ms Katherine Chan for statistical support as well as Ms Kelly Choy and Ms Evelyn Ko for technical assistance. We also thank ICU of Queen Elizabeth Hospital and ICU Outcome Monitoring Program Team of COC (ICU) of Hong Kong Hospital Authority for their support of this project.

Footnote

Conflicts of Interest: The authors have no conflicts of interest to declare.

Ethical Statement: The study was approved by the ethical committee of Queen Elizabeth Hospital (Ref: KC/KE-13-0216/ER-3). Informed consent was waived due to retrospective nature of the study.

References

1. Norris C, Jacobs P, Rapoport J, et al. ICU and non-ICU cost per day. Can J Anaesth 1995;42:192-6. [Crossref] [PubMed]
2. Hampshire PA, Welch CA, McCrossan LA, et al. Admission factors associated with hospital mortality in patients with haematological malignancy admitted to UK adult, general critical care units: a secondary analysis of the ICNARC Case Mix Programme Database. Crit Care 2009;13:R137. [Crossref] [PubMed]
3. Wunsch H, Angus DC, Harrison DA, et al. Variation in critical care services across North America and Western Europe. Crit Care Med 2008;36:2787-93, e1-9.
4. Nolan JP, Laver SR, Welch CA, et al. Outcome following admission to UK intensive care units after cardiac arrest: a secondary analysis of the ICNARC Case Mix Programme Database. Anaesthesia 2007;62:1207-16. [Crossref] [PubMed]
5. Moran JL, Bristow P, Solomon PJ, et al. Mortality and length-of-stay outcomes, 1993-2003, in the binational Australian and New Zealand intensive care adult patient database. Crit Care Med 2008;36:46-61. [Crossref] [PubMed]
6. de Vos M, Graafmans W, Keesman E, et al. Quality measurement at intensive care units: which indicators should we use? J Crit Care 2007;22:267-74. [Crossref] [PubMed]
7. Nathanson BH, Higgins TL, Teres D, et al. A revised method to assess intensive care unit clinical performance and resource utilization. Crit Care Med 2007;35:1853-62. [Crossref] [PubMed]
8. Byrick RJ, Caskennette GM. Audit of critical care: aims, uses, costs and limitations of a Canadian system. Can J Anaesth 1992;39:260-9. [Crossref] [PubMed]
9. Reddy AJ, Pappas R, Suri S, et al. Impact of throughput optimization on intensive care unit occupancy. Am J Med Qual 2015;30:317-22. [Crossref] [PubMed]
10. Mann SL, Marshall MR, Holt A, et al. Illness severity scoring for Intensive Care at Middlemore Hospital, New Zealand: past and future. N Z Med J 2010;123:47-65. [PubMed]
11. Gupta R, Arora VK. Performance evaluation of APACHE II score for an Indian patient with respiratory problems. Indian J Med Res 2004;119:273-82. [PubMed]
12. Slattery E, Basavaraju N, Ahmed S, et al. Intensive care in a general hospital: demographics, utilization and outcomes. Ir J Med Sci 2014;183:649-52. [Crossref] [PubMed]
13. Moran JL, Solomon PJ. ANZICS Centre for Outcome and Resource Evaluation of the Australian and New Zealand Intensive Care Society. Mortality and intensive care volume in ventilated patients from 1995 to 2009 in the Australian and New Zealand binational adult patient intensive care database*. Crit Care Med 2012;40:800-12. [Crossref] [PubMed]
14. Afessa B, Keegan MT, Hubmayr RD, et al. Evaluating the performance of an institution using an intensive care unit benchmark. Mayo Clin Proc 2005;80:174-80. [Crossref] [PubMed]
15. Zimmerman JE, Alzola C, Von Rueden KT. The use of benchmarking to identify top performing critical care units: a preliminary assessment of their policies and practices. J Crit Care 2003;18:76-86. [Crossref] [PubMed]
16. Breslow MJ, Badawi O. Severity scoring in the critically ill: part 1--interpretation and accuracy of outcome prediction scoring systems. Chest 2012;141:245-52. [Crossref] [PubMed]
17. Khwannimit B, Geater A. A comparison of APACHE II and SAPS II scoring systems in predicting hospital mortality in Thai adult intensive care units. J Med Assoc Thai 2007;90:643-52. [PubMed]
18. Higgins TL, Teres D, Copes WS, et al. Assessing contemporary intensive care unit outcome: an updated Mortality Probability Admission Model (MPM0-III). Crit Care Med 2007;35:827-35. [Crossref] [PubMed]
19. Zimmerman JE, Kramer AA, McNair DS, et al. Intensive care unit length of stay: Benchmarking based on Acute Physiology and Chronic Health Evaluation (APACHE) IV. Crit Care Med 2006;34:2517-29. [Crossref] [PubMed]
20. Zimmerman JE, Kramer AA, McNair DS, et al. Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today’s critically ill patients. Crit Care Med 2006;34:1297-310. [Crossref] [PubMed]
21. Zhang Z. Univariate description and bivariate statistical inference: the first step delving into data. Ann Transl Med 2016;4:91. [Crossref] [PubMed]
22. Cheng KC, Lu CL, Chung YC, et al. ICU service in Taiwan. J Intensive Care 2014;2:8. [Crossref] [PubMed]
23. Needham DM, Bronskill SE, Sibbald WJ, et al. Mechanical ventilation in Ontario, 1992-2000: incidence, survival, and hospital bed utilization of noncardiac surgery adult patients. Crit Care Med 2004;32:1504-9. [Crossref] [PubMed]
24. Hutchison CA, Crowe AV, Stevens PE, et al. Case mix, outcome and activity for patients admitted to intensive care units requiring chronic renal dialysis: a secondary analysis of the ICNARC Case Mix Programme Database. Crit Care 2007;11:R50. [Crossref] [PubMed]
25. Paul E, Bailey M, Van Lint A, et al. Performance of APACHE III over time in Australia and New Zealand: a retrospective cohort study. Anaesth Intensive Care 2012;40:980-94. [PubMed]
26. Cooke CR, Kennedy EH, Wiitala WL, et al. Despite variation in volume, Veterans Affairs hospitals show consistent outcomes among patients with non-postoperative mechanical ventilation. Crit Care Med 2012;40:2569-75. [Crossref] [PubMed]
27. Champunot R, Thawitsri T, Kamsawang N, et al. Cost effectiveness analysis of an initial ICU admission as compared to a delayed ICU admission in patients with severe sepsis or in septic shock. J Med Assoc Thai 2014;97 Suppl 1:S102-7. [PubMed]
28. Rodríguez-Carvajal M, Mora D, Doblas A, et al. Impact of the premature discharge on hospital mortality after a stay in an intensive care unit. Med Intensiva 2011;35:143-9. [PubMed]
29. Iapichino G, Gattinoni L, Radrizzani D, et al. Volume of activity and occupancy rate in intensive care units. Association with mortality. Intensive Care Med 2004;30:290-7. [Crossref] [PubMed]
30. Duke GJ. Metropolitan audit of appropriate referrals refused admission to intensive care. Anaesth Intensive Care 2004;32:702-6. [PubMed]
31. Iapichino G, Pezzi A, Borotto E, et al. Performance determinants and flexible ICU organisation. Minerva Anestesiol 2005;71:273-80. [PubMed]
32. Tan IK. APACHE II and SAPS II are poorly calibrated in a Hong Kong intensive care unit. Ann Acad Med Singapore 1998;27:318-22. [PubMed]