A large multicenter evaluation of quick Sequential Organ Failure Assessment (qSOFA) and Systemic Inflammatory Response Syndrome (SIRS) performance among hospitalized US Emergency Department patients with suspected infection

Introduction

Sepsis has caused significant consumption of healthcare resources in the United States, with at least 1.7 million adults developing sepsis and nearly 270,000 Americans died from sepsis annually (1). The mortality from sepsis was up to 38% (2-4). Sepsis has accounted for more than 20 billion dollars in annual charges, making it the single most expensive cause of hospitalization in the United States (5).

Early recognition coupled with appropriate and timely therapy is critical for sepsis management (6,7). In 1991, sepsis was defined as the Systemic Inflammatory Response Syndrome (SIRS) with infection in the presence of two or more of the following SIRS criteria: (I) a body temperature greater than 38 °C or less than 36 °C; (II) a heart rate greater than 90 beats per minute; (III) tachypnea, manifested by a respiratory rate greater than 20 breaths per minute, or hyperventilation, as indicated by a PaCO₂ of less than 32 mmHg; and (IV) an alteration in the white blood cell count, such as a count greater than 12,000/mm, a count less than 4,000/mm, or the presence of more than 10% immature neutrophils (“bands”) (8). While SIRS has become a widespread tool for screening, an evolving understanding of the pathobiology behind sepsis and the limitations of the SIRS criteria has fueled calls for refining the definition of sepsis (9,10) and better screening and prediction tools (10).

In 2015, the Third International Consensus Definitions for Sepsis and Septic Shock Task Force was convened. The revised definitions recommended discarding the term SIRS due to the overly sensitive inclusion of patients at low risk of poor outcomes and the adoption of sequential organ failure assessment (SOFA) in place of SIRS. The revised definitions focus on organ dysfunction as a marker of sepsis severity with a total SOFA score of 2 or greater meeting the criteria for sepsis (11). Sepsis-3 also introduced the quick sequential organ failure assessment (qSOFA), a simplified version of the SOFA score, for rapid bedside screening and prognostication of patients (11). The qSOFA score is determined by the presence of the following clinical criteria: (I) respiratory rate equal to or greater than 22 breaths per minute; (II) systolic blood pressure equal to or less than 100 mmHg; and (III) an alteration in mentation defined by a Glasgow coma score (GCS) less than 15 (11). The Sepsis-3 guidelines recommend qSOFA over SIRS for rapid screening purposes; however, the optimal tool for sepsis screening is still under debate (12,13).

Importance

qSOFA score could be used for the prediction of in-hospital mortality among the patients with suspected infection at Emergency Department (ED) (14). Compared to other prognostic scores, qSOFA offers similar or even better accuracy for screening of patients with sepsis for critical illness (15,16). Recognizing sepsis in ED patients, particularly those at high risk for poor outcomes, is critical to advancing sepsis care and improving patient outcomes (17). As the incidence of sepsis and resource utilization continue to increase, early and accurate identification of sepsis is essential in improving patient outcomes, resource allocation, and healthcare expenditures.

Goals of this investigation

Utilizing multicentered registry of hospitalized ED patients with suspected infection, the goal was to compare the prognostic performance of qSOFA ≥2 and SIRS ≥2 to predict clinically important outcomes using area under the receiver operating characteristic curve (AUROC) (18).

We present the following article in accordance with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting checklist (available at https://dx.doi.org/10.21037/jeccm-21-56) (19).

Methods

Study design

A subgroup analysis of data from a multicenter, observational cohort study, the United States Critical Injury and Illness Trial Group-Lung Injury Prevention Study 1 (USCIITG-LIPS 1), was performed (18).

Study setting

From March through August 2009, 22 centers (20 US and 2 non-US hospitals) enrolled patients, admitted from EDs and hospitalized for elective surgery, with at least one or more a priori defined conditions predisposing to acute lung injury (ALI), as previously described (18).

Patient and public involvement

This study is a subgroup analysis of a larger multi-center cohort study, so patient or the public were not involved in this study.

Selection of participants

Subjects from the LIPS 1 study were considered appropriate for inclusion if they were 18 years or older, admitted to US academic and community acute care hospitals EDs, and diagnosed with sepsis or pneumonia (9). Pneumonia patients were included if chest radiographs demonstrated new or progressive infiltrates, consolidation, cavitation, or pleural effusion and either presence of new onset or change in character of purulent sputum change. Positive cultures were used when available (18). Patients were excluded if ALI was present at initial assessment, transferred from another institution from the inpatient setting, died in the ED, admitted for palliative, hospice care, or elective surgery, or readmitted during the study period. In addition, subjects were excluded if the GCS or vital signs were not recorded. A study flow diagram is illustrated in Figure S1. The SIRS definition of sepsis was applied and recorded as part of the original study criteria. The qSOFA score was applied retrospectively to all patients with suspected or diagnosed infection.

Data collection and processing

Baseline characteristics, including demographics, comorbidities, and clinical variables, were collected during the first 6 hours of initial ED evaluation. Prior to study initiation at each site, investigators and study coordinators received structured training. The principal investigators from each site were responsible for data collection, data entry, and quality control. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Institutional Review Board of Mass General Brigham (NO. IRB00012706) and individual consent for this retrospective analysis was waived.

Outcome measures

The primary outcome was hospital mortality among patients with suspected or diagnosed infection. Secondary outcomes included a composite of hospital mortality or intensive care unit (ICU) length of stay (LOS) ≥3 days, ICU LOS ≥3 days alone, overall ICU utilization, invasive mechanical ventilation, non-invasive ventilation, vasopressor use, and hemodialysis secondary to acute renal failure.

Statistical analysis

Baseline demographics and clinical characteristics were examined using one-way analysis of variance (ANOVA) and Kruskal-Wallis tests for normally and non-normally distributed continuous variables, respectively, and chi-square or Fisher’s exact tests for categorical variables. We assessed the association between qSOFA ≥2 and SIRS ≥2 for each clinical outcome using generalized linear mixed-effects regression models, accounting for the correlation among ED patients from the same study site. Additionally, we assessed the prognostic performance of qSOFA ≥2 and SIRS ≥2 criteria to predict the primary and secondary clinical outcomes using AUROC. Odds ratios (ORs) and AUROC were reported along with 95% confidence intervals. AUROC comparisons were made using the DeLong test (20). We additionally calculated sensitivity, specificity, positive predictive value (PPV), and negative predictive values (NPV) for each cut-off. All significance tests were two-sided, with a P value <0.05 considered statistically significant. All of the statistical analyses were performed using Statistical Analysis Software (SAS) version 9.4 (SAS Institute Inc., Cary, North Carolina, USA).

Results

Characteristics of study population

Between March and August 2009, 5,584 patients with at least one predisposing condition for ALI at the time of hospital ED evaluation or admission for elective surgery were enrolled. After exclusion criteria were applied, the final cohort comprised of 1,689 patients who met inclusion criteria for presumed or documented infection and complete data to calculate the qSOFA score (Figure S1). The median age of the study population was 57 years [interquartile range (IQR), 45, 71 years]. Males accounted for 50.7% of the cohort (P=0.012) (Table 1).

Main results

Overall, 22% (372/1,689) of patients met qSOFA ≥2 and 90% (1,519/1,689) met SIRS ≥2. The distribution of qSOFA elements among those with qSOFA ≥2 (n=372) for GCS <15, SBP <100 mmHg, and respiratory rate ≥22 were 45.4%, 80.9%, and 89.8% respectively. SIRS elements were distributed among SIRS ≥2 (n=1,519) for respiratory rate >20 or PaCO₂ >32 mmHg, heart rate >90, temperature >38° or <36°, white blood cell count >12 or <4 (×10⁹/L) with 52.4%, 83.9%, 46.2%, and 65.2%, respectively. Patients with chronic diseases were more likely to be classified as qSOFA ≥2 vs. qSOFA <2 if they had chronic obstructive pulmonary disease (COPD) (17.7% vs. 12.7%, P=0.017) or congestive heart failure [CHF New York Heart Association (NYHA) Class IV] (7.3% vs. 4.3%, P=0.03). Those with immunosuppression were more likely to meet SIRS ≥2 vs. SIRS <2, (17% vs. 8.8%, P=0.006) as were patients with cirrhosis (3.5% vs. 0%, P=0.005). On the other hand, patients were less likely to have a history of diabetes mellitus if they met SIRS ≥2 vs. SIRS <2 criteria (29.0% vs. 38.2%, P=0.017). Compared to patients with qSOFA <2, those with qSOFA ≥2 were more likely to have been diagnosed with shock (28.0% vs. 3.0%, P=0.001) and higher median Acute Physiology and Chronic Health Evaluation (APACHE) II score (16 vs. 10, P<0.001).

Overall, 14.2% (239/1,689) of patients had GCS <15 (P<0.001). As expected, variables inherent to the definitions were different. The clinical characteristics of the cohort are summarized in Table 1.

Outcomes

Overall hospital mortality rate was 5.2% (88/1,689). OR of qSOFA ≥2 compared to qSOFA <2 for death was 4.61 (95% CI: 2.96–7.19, P=0.001) and was highest for vasopressor use at 6.92 (95% CI: 4.70–10.19) followed by composite outcome of death and ICU LOS ≥3 at 5.31 (95% CI: 4.01–7.02). Significant ORs for all other outcomes were higher for qSOFA ≥2 compared to SIRS ≥2 (Tables 2,3).

We further analyzed the demographic, clinical characteristics, vital signs, and laboratory values of all patients who had either qSOFA ≥2 or SIRS ≥2 and died. Interestingly, patients with qSOFA ≥2 had higher alcohol consumption (21% vs. 3%, P=0.002), lower median diastolic blood pressure (49.5 vs. 64.5 mmHg, P=0.002) and lower HCO₃ (20.5 vs. 24.7 mEq/L, P=0.01) as compared to patients with SIRS ≥2. Those who had SIRS ≥2 and died were more likely to have CHF NYHA Class IV (7% vs. 1%, P=0.04) or been exposed to systemic steroids (14% vs. 6%, P=0.03) (Tables S1,S2).

Predictive performance

Overall, the AUROC for qSOFA ≥2 was greater than SIRS ≥2. AUROC for death was higher for qSOFA ≥2 (0.66, 95% CI: 0.60–0.73) vs. SIRS ≥2 (0.55, 95% CI: 0.49–0.61).

qSOFA ≥2 had the strongest predictability for both vasopressor use and composite outcome of death and ICU LOS ≥3 days, with an adequate AUROC of 0.77 (95% CI: 0.73–0.81) and 0.77 (95% CI: 0.74–0.80) respectively. qSOFA ≥2 consistently demonstrated significantly better predictive performance than SIRS ≥2 for all outcomes, except for non-invasive ventilation and hemodialysis. When combining the two criteria, SIRS ≥2 did not significantly improve the performance of qSOFA ≥2 in predicting outcomes (Table 4, Figure 1, Figures S2,S3).

Figure 1 AUROC for death. AUROC, area under the receiver operating characteristic curve; qSOFA, quick Sequential Organ Failure Assessment; SIRS, Systemic Inflammatory Response Syndrome.

qSOFA ≥2 was more specific for all outcomes measured, while SIRS ≥2 was more sensitive. Specificity for qSOFA ≥2 was four times greater than SIRS ≥2, ranging from 79% to 87% whereas sensitivity for SIRS ≥2 was twice that of qSOFA ≥2 at an average of 88%. PPVs for qSOFA ≥2 and SIRS ≥2 were highest for ICU utilization at 57% and 31%, respectively. NPVs ranged from 80% to 97% for both qSOFA ≥2 and SIRS ≥2 (Table S3).

Discussion

Advantages of the qSOFA score over the SIRS criteria include its simplicity and the ability to calculate the score without laboratory data. Despite these advantages, the value of the qSOFA score as a screening tool to identify patients presenting with sepsis in non-ICU settings was called into question (21,22). ED LOS is also known to play a role in the survival of patients with sepsis requiring ICU admission (23). In this study, we assessed the predictive ability of the qSOFA ≥2 criteria to identify ED patients with suspected infection at risk for poor outcomes. qSOFA ≥2 was better than SIRS ≥2 in predicting in-hospital death and the composite outcome of death or ICU LOS ≥3 days. qSOFA ≥2 was a significant predictor for ICU utilization, invasive and non-invasive ventilation vasopressor use, and hemodialysis as compared to SIRS ≥2. However, despite significant gains in accuracy, qSOFA ≥2 possessed a poor predictor for death with AUROC 0.66. In contrast, qSOFA ≥2 had an adequate AUROC of 0.77 for the composite outcome of death or ICU LOS ≥3 days.

In this ED cohort, SIRS ≥2 was found to have greater sensitivity for in-hospital mortality compared with qSOFA ≥2, potentially lending credence to doubts about qSOFA, though an inverse trend was noted regarding the specificity of the two tools, with qSOFA ≥2 outperforming SIRS ≥2. These trends have been noted in other ED cohorts, and together lend strong support and voiced by many in the emergency medicine community, that qSOFA ≥2 has inadequate sensitivity to be utilized in a screening capacity (21,22). For predicting poor outcomes of patients with sepsis in the ED, qSOFA may be superior to SIRS, but the sensitivity of qSOFA is of great concern (24,25). Adoption of qSOFA ≥2 as a triage screening tool for sepsis would likely expose EDs to miss many patients at risk for poor outcomes. Utilizing qSOFA ≥2 criteria as an initial screening tool for triggering ED diagnostic and treatment pathway in sepsis is not supported by our findings.

The qSOFA ≥2 vs. SIRS ≥2 controversy potentially represents a form of discord between construct validity and criterion/predictive validity. This is of significance because sepsis is a clinical syndrome versus a discrete disease with a singular pathology. Given that qSOFA was shown to be superior in predicting poor outcomes, applying the qSOFA score to ED patients with suspected infection who are more likely to develop sepsis and organ dysfunction may be valuable in determining floor vs. ICU admission (21,22). Similar to our findings, several studies documented a modest increase in accuracy of qSOFA vs. SIRS in terms of predicting mortality (21,22). Given the lack of sensitivity, yet adequate to good prediction of hospital death or ICU ≥3 days and ICU utilization, qSOFA ≥2 may be a useful tool for risk stratification of ED patients with sepsis at risk for deterioration. Whereby ICU resources are at a premium in institutions, qSOFA ≥2 may be used as a tool to standardize the most appropriate inpatient designation for the sepsis patient in the ED. In addition, qSOFA ≥2 could conceivably be used in the pre-hospital arena in austere or remote environments, and meeting qSOFA criteria could trigger the transportation of a suspected sepsis patient at risk of deterioration to a center with higher levels of care, capable of managing high risk sepsis patient (25).

To the best of our knowledge, this study represents the largest US multi-center study to examine the performance of qSOFA ≥2 as compared to SIRS ≥2 among hospitalized ED patients with suspected infection. As there are more than 500,000 ED patients presenting with sepsis annually, emergency physicians need both good screening and prognosticating tools in order to deliver the most appropriate care to their patients in a timely fashion. Until then, the Surviving Sepsis Campaign (SSC) international guidelines for the management of sepsis and septic shock will continue to state, “we recommend that hospitals and hospital systems have a performance improvement program for sepsis, including sepsis screening for acutely ill, high-risk patients (17).”

Limitations

Our study has several limitations. First, the cohort did not include all-comers to the ED with suspected or diagnosed infection and was limited to those patients with either identified pneumonia or sepsis at the time of enrollment in the original LIPS 1 cohort. Sepsis was defined with clinical symptoms and SIRS ≥2 criteria, which could cause bias from not considering non-pneumonia patients with qSOFA ≥2. Also, the absence of data regarding time-to-onset of the SIRS and qSOFA criteria and the longitudinal characteristics of sepsis could affect criteria utility (26). For instance, it is plausible that tachycardia manifested earlier than altered mental status in sepsis, which may increase the utility of SIRS criteria over qSOFA criteria in terms of screening ED patients with suspected infection at risk for poor outcomes from sepsis. Lastly, with the SIRS criteria and consequent sepsis definition available at the time of the enrollment of these patients, it is difficult to establish whether clinical management was altered on the basis of that ascertainment conferring relatively better outcomes among patients with SIRS ≥2 compared to those with qSOFA ≥2.

Conclusions

Among this cohort of ED patients with suspected infection, qSOFA ≥2 was a better predictor compared to SIRS ≥2 for hospital mortality, composite outcome of death or ICU LOS ≥3 days, ICU utilization, invasive ventilation, and vasopressor use. However, the performance of qSOFA ≥2 by AUROC for hospital mortality was modest to adequate for the composite outcome of death or ICU LOS ≥3 days. Moreover, qSOFA ≥2 fell short in sensitivity compared to SIRS ≥2. Neither tool appears sufficient for independent use in the prognostication of hospitalized ED patients with suspected infection. The findings from this study suggest that further multicenter prospective trials are warranted to examine the utility of qSOFA as a screening tool.

Acknowledgments

This study was presented in part at the Society of Critical Care Medicines 46th Annual Conference in Hawaii, January 2017.

Funding: The efforts of this study were supported by the internal funding at the University of Florida Departments of Emergency Medicine and Medicine, Division of Nephrology to Marie-Carmelle Elie-Turenne.

Footnote

Reporting Checklist: The authors have completed the STROBE reporting checklist. Available at https://dx.doi.org/10.21037/jeccm-21-56

Data Sharing Statement: Available at https://dx.doi.org/10.21037/jeccm-21-56

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://dx.doi.org/10.21037/jeccm-21-56). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Institutional Review Board of Mass General Brigham (NO. IRB00012706) and individual consent for this retrospective analysis was waived.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Prevention CfDCa. Sepsis Factsheet 2020. Available online: https://www.cdc.gov/sepsis/education/hcp-resources.html
2. Elixhauser A, Friedman B, Stranges E. Septicemia in U.S. Hospitals, 2009: Statistical Brief #122. Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Rockville (MD): Agency for Healthcare Research and Quality (US), 2006.
3. Angus DC, Linde-Zwirble WT, Lidicker J, et al. Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Crit Care Med 2001;29:1303-10. [Crossref] [PubMed]
4. Bhattacharjee P, Edelson DP, Churpek MM. Identifying Patients With Sepsis on the Hospital Wards. Chest 2017;151:898-907. [Crossref] [PubMed]
5. Torio CM, Andrews RM. National Inpatient Hospital Costs: The Most Expensive Conditions by Payer, 2011: Statistical Brief #160. Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Rockville (MD): Agency for Healthcare Research and Quality (US), 2006.
6. Ferrer R, Martin-Loeches I, Phillips G, et al. Empiric antibiotic treatment reduces mortality in severe sepsis and septic shock from the first hour: results from a guideline-based performance improvement program. Crit Care Med 2014;42:1749-55. [Crossref] [PubMed]
7. Stevenson EK, Rubenstein AR, Radin GT, et al. Two decades of mortality trends among patients with severe sepsis: a comparative meta-analysis*. Crit Care Med 2014;42:625-31. [Crossref] [PubMed]
8. Bone RC, Balk RA, Cerra FB, et al. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine. Chest 1992;101:1644-55. [Crossref] [PubMed]
9. Levy MM, Fink MP, Marshall JC, et al. 2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference. Crit Care Med 2003;31:1250-6. [Crossref] [PubMed]
10. Singer M, Deutschman CS, Seymour CW, et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA 2016;315:801-10. [Crossref] [PubMed]
11. Usman OA, Usman AA, Ward MA. Comparison of SIRS, qSOFA, and NEWS for the early identification of sepsis in the Emergency Department. Am J Emerg Med 2019;37:1490-7. [Crossref] [PubMed]
12. Choi A, Park YS, Shin TG, et al. Prognostic performance of disease severity scores in patients with septic shock presenting to the emergency department. Am J Emerg Med 2019;37:1054-9. [Crossref] [PubMed]
13. Canet E, Taylor DM, Khor R, et al. qSOFA as predictor of mortality and prolonged ICU admission in Emergency Department patients with suspected infection. J Crit Care 2018;48:118-23. [Crossref] [PubMed]
14. Koyama S, Yamaguchi Y, Gibo K, et al. Use of prehospital qSOFA in predicting in-hospital mortality in patients with suspected infection: A retrospective cohort study. PLoS One 2019;14:e0216560 [Crossref] [PubMed]
15. Martino IF, Figgiaconi V, Seminari E, et al. The role of qSOFA compared to other prognostic scores in septic patients upon admission to the emergency department. Eur J Intern Med 2018;53:e11-3. [Crossref] [PubMed]
16. Rodriguez RM, Greenwood JC, Nuckton TJ, et al. Comparison of qSOFA with current emergency department tools for screening of patients with sepsis for critical illness. Emerg Med J 2018;35:350-6. [Crossref] [PubMed]
17. Rhodes A, Evans LE, Alhazzani W, et al. Surviving Sepsis Campaign: International Guidelines for Management of Sepsis and Septic Shock: 2016. Intensive Care Med 2017;43:304-77. [Crossref] [PubMed]
18. Gajic O, Dabbagh O, Park PK, et al. Early identification of patients at risk of acute lung injury: evaluation of lung injury prediction score in a multicenter cohort study. Am J Respir Crit Care Med 2011;183:462-70. [Crossref] [PubMed]
19. von Elm E, Altman DG, Egger M, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. Int J Surg 2014;12:1495-9. [Crossref] [PubMed]
20. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988;44:837-45. [Crossref] [PubMed]
21. Williams JM, Greenslade JH, McKenzie JV, et al. Systemic Inflammatory Response Syndrome, Quick Sequential Organ Function Assessment, and Organ Dysfunction: Insights From a Prospective Database of ED Patients With Infection. Chest 2017;151:586-96. [Crossref] [PubMed]
22. Churpek MM, Snyder A, Han X, et al. Quick Sepsis-related Organ Failure Assessment, Systemic Inflammatory Response Syndrome, and Early Warning Scores for Detecting Clinical Deterioration in Infected Patients outside the Intensive Care Unit. Am J Respir Crit Care Med 2017;195:906-11. [Crossref] [PubMed]
23. Zhang Z, Bokhari F, Guo Y, et al. Prolonged length of stay in the emergency department and increased risk of hospital mortality in patients with sepsis requiring ICU admission. Emerg Med J 2019;36:82-7. [PubMed]
24. Park HK, Kim WY, Kim MC, et al. Quick sequential organ failure assessment compared to systemic inflammatory response syndrome for predicting sepsis in emergency department. J Crit Care 2017;42:12-7. [Crossref] [PubMed]
25. Askim Å, Moser F, Gustad LT, et al. Poor performance of quick-SOFA (qSOFA) score in predicting severe sepsis and mortality - a prospective study of patients admitted with infection to the emergency department. Scand J Trauma Resusc Emerg Med 2017;25:56. [Crossref] [PubMed]
26. Zhang Z, Ho KM, Gu H, et al. Defining persistent critical illness based on growth trajectories in patients with sepsis. Crit Care 2020;24:57. [Crossref] [PubMed]