Non-applicability of a validated predictive model for intensive care admission and death of COVID-19 patients in a secondary care hospital in Belgium

Introduction

The novel coronavirus disease 2019 (COVID-19) presents an important and urgent threat to global health. Since the outbreak in early December 2019 in the Hubei Province of China, the number of patients confirmed to have the disease has exceeded 123,321,541 cases in more than 198 countries (up to 22 March 2021), and the number of people infected is probably much higher. More than 2,716,990 people have died from COVID-19 infection (up to 22 March 2021) (1). Despite public health responses aimed at containing the disease and delaying the spread, several countries including Belgium have been confronted with a critical care crisis. Outbreaks lead to important increases in the demand for hospital beds and shortage of medical equipment. The adequacy of resources to treat infected cases is therefore a growing public health concern.

The spread of COVID-19 has been heterogeneous, resulting in some regions having sporadic transmission and relatively few hospitalized patients with COVID-19 and others having community transmission that has led to overwhelming numbers of severe cases. The latter is the case for Belgium which has one of the highest incidence and death rates in the world [22,707 deaths, corresponding to 1,976 deaths per million inhabitants (up to 22 March 2021)] (1). South of Brussels, in a region called Brabant Wallon in Belgium, where Clinique Saint-Pierre Ottignies is located, healthcare delivery has been compromised by critical resource constraints in diagnostic testing, hospital beds, ventilators, and healthcare workers. Optimal allocation of resources has led us to build easily implemented clinical indicators in order to best guide patients at the time of admission and avoid futile treatments in intensive care units (ICUs).

Time is a huge constraint in the management of COVID-19 patients during the pandemic. We excluded from our study all biological variables that were not available within a few hours after blood sampling. In doing so, we are in line with the reality of most non-tertiary care hospitals in Europe.

We present the following article in accordance with the MDAR reporting checklist (available at http://dx.doi.org/10.21037/jeccm-20-173).

Role of radiology in patient management

Thoracic imaging with chest radiography (CXR) and thoracic computed tomography (TCT) are key tools for pulmonary disease diagnosis and management, but their role in the management of COVID-19 has been sparsely considered within the multivariable context of the severity of respiratory disease, pre-test probability, risk factors for disease progression, and critical resource constraints. In a recent consensus article (2), a multidisciplinary panel comprised principally of radiologists and pulmonologists from 10 countries with experience managing COVID-19 patients across a spectrum of healthcare environments has recently evaluated the utility of imaging within three scenarios representing varying risk factors, community conditions, and resource constraints. The main recommendations of this panel of experts are:

• Imaging is not routinely indicated as a screening test for COVID-19 in asymptomatic individuals;
• Imaging is not indicated for patients with mild features of COVID-19 unless they are at risk for disease progression;
• Imaging is indicated for patients with moderate to severe features of COVID-19 regardless of COVID-19 test results;
• Imaging is indicated for patients with COVID-19 and evidence of worsening respiratory status;
• In a resource-constrained environment where access to CT is limited, CXR may be preferred for patients with COVID-19 unless features of respiratory worsening warrant the use of CT.

These recommendations, quite sensible at first sight, are not confirmed by any study and are challenged by our clinical experience. A recent review of the radiological manifestations of COVID-19 gives no clear-cut conclusions (3) and the association between radiological findings and outcome remain elusive. We consequently included two radiological variables within the features of interest and checked their statistical significance.

Clinical score in the literature

A recent publication (4) provides a review and critical appraisal of published and preprint reports of prediction models for diagnosing patients with suspected infection, for prognosis of COVID-19 patients, and for detecting people in the general population at risk of being admitted to hospital for COVID-19 pneumonia. In this review, 31 prediction models were included. The conclusions of the authors are clear-cut: all studies are rated at high risk of bias, mostly because of non-representative selection of control patients, exclusion of patients who had not experienced the event of interest by the end of the study, and high risk of model overfitting. Reporting quality varied substantially between studies. Most reports did not include a description of the study population or intended use of the models, and calibration of predictions was rarely assessed. Authors recommended that further studies should adhere to the TRIPOD (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) reporting guideline (5).

Among the clinical scores available in the literature, one of them drew our attention (6). Having been recently published in JAMA Internal Medicine, it addresses the same issues as our study, but was conducted on a much larger scale. According to the authors, the development cohort included 1,590 patients. The mean (SD) age of patients in this cohort was 48.9 (15.7) years; 904 (57.3%) were men. The validation cohort included 710 patients with a mean (SD) age of 48.2 (15.2) years, and 382 (53.8%) were men and 172 (24.2%). From 72 potential predictors, 10 variables were independent predictive factors and were included in the risk score: chest radiographic abnormality (OR, 3.39; 95% CI, 2.14–5.38), age (OR, 1.03; 95% CI, 1.01–1.05), hemoptysis (OR, 4.53; 95% CI, 1.36–15.15), dyspnea (OR, 1.88; 95% CI, 1.18–3.01), unconsciousness (OR, 4.71; 95% CI, 1.39–15.98), number of comorbidities (OR, 1.60; 95% CI, 1.27–2.00), cancer history (OR, 4.07; 95% CI, 1.23–13.43), neutrophil-lymphocyte ratio (NLR) (OR, 1.06; 95%CI, 1.02-1.10), lactate dehydrogenase (LDH) (OR, 1.002; 95% CI, 1.001–1.004) and direct bilirubin (OR, 1.15; 95% CI, 1.06–1.24). The area under the receiver operating characteristic (ROC) curve (AUC) in the development cohort was 0.88 (95% CI, 0.85–0.91) and the AUC in the validation cohort was 0.88 (95% CI, 0.84–0.93). The score has been translated into an online risk calculator that is freely available (http://118.126.104.170/). The fact that the clinical score was validated on an independent large population gives some credit to its applicability on a large scale. We decided then to include this score in our study and, for each of our COVID-19 patients, we calculated the clinical risk score at time d₀ (day of admission to the emergency ward) using the freely available calculator. We refer to this clinical risk score as Liang score (6). Liang score prediction accuracy is used as a benchmark of our own search of a predictive model. However, in contrast to Liang approach, we have distinguished two outcomes that cannot be confounded: admission to ICU and death. Our research questions are the following:

• Is it possible to predict admission of COVID-19 patients to ICU with routine and quickly available clinical, biological and radiological variables? What is the prediction error of the selected model(s)?
• Is it possible to predict death among COVID-19 patients with routine and quickly available clinical, biological and radiological variables, in order to avoid unnecessary treatment and waste of precious resources? What is its prediction error?

Methods

We conducted a retrospective cohort study of 66 patients with known COVID-19 disease, from March 10^th to May 16^th 2020 in Clinique Saint-Pierre Ottignies in Belgium. Twenty patients were admitted in ICU and 44 in Medicine Department. Sixty-three patients were positive for a nasopharyngeal RT-PCR SARS-CoV-2 test. One patient included in the study was negative for RT-PCR SARS-CoV-2 but had an IgA and IgG ELISA-test positive for SARS-CoV-2 and ground-glass opacities on the chest X-ray. Patients characteristics are displayed in Table 1. All patients were followed from their admission at the emergency ward until they get out of the hospital or until their death. No patient was excluded from the cohort. The follow-up ended when patients leaved the hospital or died. The starting date of accrual and the end-date of accrual were reported for each patient. Clinique Saint-Pierre Ottignies is a 425-bed regional general hospital with a capacity of 15 intensive care beds, which was increased to 25 beds during the pandemic. It has a mission of para-university training of junior medical specialists. It covers a catchment area of circa 400,000 patients, in the region of Brabant Wallon (Wallonia, Belgium).

Table 1 Patient characteristics
Full table

The study was conducted in accordance with Declaration of Helsinki (as revised in 2013). The study was approved by the Institutional Ethics Committee of our hospital “Ethics Committee Clinique Saint-Pierre Ottignies” (Ethical Approval ID 2020-13). The euract number of the study is B0432021000000. Individual consent for this retrospective analysis was waived.

Criteria of admission to the ICU

Twenty-two patients with COVID-19 were admitted to the ICU. For 19 patients the reason was a respiratory failure defined as (I) ambient oxygen saturation (SpO₂) <88% with nasal cannula oxygen therapy >5 L/min; (II) PaO₂ <50 mmHg and/or a ratio PaO₂/FiO₂ <150; (III) respiratory rate >40/min. For one patient, the reason was a post-traumatic cerebral hemorrhage, associated with altered neurological status (defined as Glasgow coma scale <8/15) requiring mechanical ventilation in order to protect the airway. For another one, the cause was the postoperative management of an empyema drainage developed in a context of bacterial pneumonia complicating a SARS-CoV-2 virus infection. A third patient presented a status epilepticus in the context of probable alcohol withdrawal.

Criteria of non-admission to the ICU

Clinique Saint-Pierre Ottignies has long-proven guidelines for admission to ICU. Given the nature of the health emergency, the principle of distributive justice (7) was applied and each patient admitted to the emergency department was immediately classified as “eligible for intensive care” or “not eligible for intensive care”, taking into account his or her previous history and quality of life. The criteria for ineligibility were: (I) presence of a prior incurable disease; (II) limitation of functional autonomy; and (III) advanced dementia. Two patients had criteria for intensive care hospitalization upon admission to the emergency room, the other patients were first hospitalized in a non-ICU.

The patients admitted to intensive care all had the clinical criteria mentioned above, with a pre-established maximalist therapeutic plan. Other patients with the same clinical criteria but with a care plan with therapeutic limitations were not admitted. These therapeutic projects were discussed collegially between medical specialties issued from emergency department, internal medicine and critical unit. The advantage of a simple predictive score upon admission could be useful in decision-making regarding a therapeutic plan.

Assessment of lung injury

A senior radiologist analyzed TCT according to two criteria: percentage of lung injury (% LI) (continuous variable from 0 to 100%) and density of lung injury (DLI) (factor variable with three grades: 1= light density; 2= moderate density; 3= high density).

Data collection

Data collection tried to adhere as tightly as possible to the TRIPOD Adherence extraction form (https://www.tripod-statement.org/wp-content/uploads/2020/03/TRIPOD-Adherence-assessment-form_V-2018_12.pdf). At their admission, patients were questioned about their usual medication and their health condition. The body mass index was computed. Collected variables are the following: age, gender, ethnic group, weight, body mass index, number of days with symptoms before hospitalization, asthenia, pyrexia, dyspnea, chest pain, a specific digestive symptoms, anosmia, ageusia, confusion, travel or contact <1 month, cigarette consumption (Y or N), hypertension, diabetes, mental status (depression), angiotensin-converting-enzyme inhibitors, angiotensin II receptor antagonists, non-steroidal anti-inflammatory drugs, immunosuppressive drugs, SpO₂ (%), TCT % of lung injury, TCT DLI, blood type, white blood cells, neutrophils, lymphocytes, blood platelets, fibrinogen, ferritin, triglycerides, LDH, troponin, C-reactive protein (CRP), NLR, lymphocyte-to-CRP ratio, bilirubin and lactates. The dates of admission to ICU and death were recorded.

In the ICU, the use of chloroquine, hydroxychloroquine, azithromycin, clarithromycin, remdesevir and antibacterial antibiotics (piperacillin-tazobactam, meropenem, ciprofloxacin, ceftazidime, amoxicillin clavulanate) was recorded on a daily basis.

As mentioned previously, the clinical risk score developed by Liang et al. was computed retrospectively for all COVID-19 patients at their admission to the emergency ward. This clinical risk score was compared to our own models.

Statistical analysis

Theoretically, every variable collected in the study could be a candidate predictor in a logistic regression. Nevertheless, in order to reduce the risk of false positive findings and improve model performance, the events per variable (EPV) must be considered. A rule of thumb of 10 individuals per event is commonly applied (8). However, recent studies have shown that EPV does not have a strong relation with metrics of predictive performance, and is not an appropriate criterion for binary prediction model development studies (9). According to reference (10), predictive performance problems are fairly frequent with 2–4 EPV, uncommon with 5–9 EPV, and still observed with 10–16 EPV. The rule of thumb of 10 individuals per event can then be relaxed. In our study, EPV ranges from 7 to 20. This EPV rule implies that only models made of less than four variables should be selected and tested. The number of collected variables being 40, the number of possible predictive models is extremely large, approximately 2⁴⁰ (11). Consequently, testing all of these models is not computationally feasible and a preselection of potential predictive models was performed using a recently developed method (12). This selection procedure picks out models with the lowest Prediction Error computed by 10 folds cross-validation inside a 5% confidence range. Therefore, these models have equivalent predictive power. This procedure has the benefit of selecting a set of small dimension models, thus complying with the EPV rule of around 10. Four models had the lowest prediction error for ICU admission computed by 10 folds cross-validation: (I) LDH, (II) LDH + sex, (III) LDH + sex + venous lactate, and (IV) respiratory impairment score (a combination of % LI, DLI, and SpO₂). Similarly, three models were suited to predict death: (I) NLR + tobacco; (II) Liang score + NLR + tobacco; and (III) respiratory impairment score. Not surprisingly, LDH, alone or in combination, is included in three out of four models predicting ICU admission, supporting recent literature findings (13,14).

Logistic regressions were used in our study in order to address research questions based on binary outcomes. The multivariate analysis estimates coefficients (for example, log odds or hazard ratios) for each predictor included in the final model and adjusts them with respect to the other predictors in the model. The coefficients quantify the contribution of each predictor to the outcome risk estimation (15). The caveats to consider when assessing the results of a logistic regression analysis are well explained in (16). The binomial family logit function and the maximum likelihood approach were used to compute the regression coefficients. The coefficients are then equivalent to the relative risk of the outcome in the exposed group (COVID-19 patients). All analysis were performed using the R software version 4.0.1 (https://cran.r-project.org/).

Results

Admission to ICU and death are the relevant end-points of our study, and they have been handled separately. Pathophysiology of COVID-19 is complex (17) and the process leading to death merges intrinsic factors [endothelial dysfunction (18), thrombotic complications (19), respiratory distress syndrome (20), renal failure (21), cardiovascular collapse (22), etc.] and extrinsic factors to the disease [patient self-inflicted lung injury (23), central nervous system impairment due to long-term sedation (24), etc.]. Moreover, admission to ICU and death are outcomes that are deeply different in nature. This point will be discussed further on. For both outcomes, Liang score is the benchmark to which any predictive score will be compared to.

Admission to ICU

Predictive power of Liang score on the Belgian cohort was computed and used as a reference standard for our selected models. The predictive performances of competing models are shown in Table 2. As mentioned previously, these models were selected in order to keep the EPV in the 7 to 10 range. If the sensitivity of Liang score value is high, its specificity is very low (0.10). Its overall accuracy is lower than 70% and it is outperformed by simpler models (Table 2). Liang score regression coefficient is close to zero and above the significance threshold of 5% (Table 3). Liang score predicts 2 admissions to ICU out of 20 patients and cannot reliably be used as a clinical tool in the emergency ward. In the wake of this disappointing result, we computed our own predictive scores, with a minimum number of covariates in order to keep the EPV in the 7 to 10 range. As mentioned previously, we carried out a pre-selection of models using the SWAG method (12). Four models stand out from the pack with the lowest cross-validation error: (I) LDH; (II) LDH + sex; (III) LDH + sex + lactate; and (IV) respiratory impairment. The respiratory impairment score is built out of three variables: % LI (continuous variable from 0 to 100%), DLI (factor variable with three grades: 1= light density; 2= moderate density; 3= high density), and SpO₂. The overall performances of these models are displayed in Table 2 and the estimated parameters for these models are shown in Tables 4-7.

Table 2 Predictive performances of competing models regarding admission to ICU
Full table

Table 3 Liang score logistic regression coefficients (admission to ICU)
Full table

Table 4 LDH score logistic regression coefficients (admission to ICU)
Full table

Table 5 (LDH + sex) score logistic regression coefficients (admission to ICU)
Full table

Table 6 (LDH + sex + venous lactate) score logistic regression coefficients (admission to ICU)
Full table

Table 7 Respiratory impairment score logistic regression coefficients (admission to ICU)
Full table

It appears that no score is quite satisfactory to predict admission to ICU. All scores that were tested are flawed by a lack of specificity and cannot be reliably used in the emergency ward. However, the absence of significant results does not mean that no valuable information can be extracted. Indeed, there are three kinds of useful information that can be learned from these analyses.

First, the predictive score designed by Liang et al. on a development cohort of 1,590 patients and validated over 710 patients in China appears to perform poorly on a European population with overall accuracy lower than 70% (95% CI: 0.5431–0.7841), and specificity around 10%. This surprising result requires additional clarification. At first sight, the main reason of Liang score poor performance on the Belgian cohort lies in the difference of end-points. Based on the American Thoracic Society guidelines for community-acquired pneumonia (25), Liang et al. defined critical COVID-19 illness as a composite of admission to the ICU, invasive ventilation, or death. They adopted this composite end-point for the reason that admission to ICU, invasive ventilation, and death are serious outcomes of COVID-19 that have been adopted in previous studies to assess the severity of other serious infectious diseases. However, admission to ICU, invasive ventilation, and death are end-points whose nature is quite different. Admission to ICU and invasive ventilation depend on many extrinsic factors such as availability of intensive care beds, hospital care policy, and fair allocations of resources. These factors can vary widely over time. At the peak of the pandemic, overcrowding of ICU beds can divert patients to lighter care structures. On the other hand, death could be seen as a more “objective” end-point, less prone to health care policy fluctuations. The difference of end-points may then explain the lack of predictive power for Liang score. Another reason may lie in the genetics of population. This point deserves further research.

The second information is that respiratory impairment performs better than Liang score but can predict only 7 admissions to ICU out of 20. This finding confirms the fact that COVID-19 is much more than a respiratory distress syndrome (26); it is essentially a multiple endothelial dysfunction (27) with de novo angiogenesis and thrombosis (28).

The third information is that simple and affordable variables such as LDH, LDH + sex, or LDH + sex + venous lactate have a very good sensitivity and an acceptable specificity. From the logistic regression coefficients, it appears that a serum LDH value above 579 U/L is a good predictor of admission to the ICU (Table 4). In addition, the odds ratio for gender-related risk for ICU admission is 1.71 higher for men than for women (Table 5). This is quite in line with our experience. In addition, a serum lactate value greater than 3 mmol/L is also a good severity predictor (Table 6).

In order to assess the superiority of (LDH + sex), (LDH + sex + lactate) and (respiratory impairment) models over Liang score, we draw the respective ROC curves (Figure 1) of the models and performed a Delong test (29) to compare the relevant AUCs (Table 8). ROC curves and Delong test (29) show that (LDH +sex) and (LDH + sex + lactate) have statistically significant greater AUC than Liang score. At the opposite, respiratory impairment score does not perform better than Liang score (Table 8).

Figure 1 ROC curve of the two most relevant scores (LDH + sex and LDH + sex + lactate) compared to Liang score. ROC, receiver operating characteristic; LDH, lactate dehydrogenase.

Table 8 Delong test comparing AUC of the four more relevant predictive models for ICU admission
Full table

In conclusion, our results suggest that male gender, a LDH value above 579 U/L and a blood lactate value above 3 mmol/L are good predictors for ICU admission.

Death

Mutatis mutandis, we used the same methodology to preselect models having the highest accuracy to predict death. The SWAG method preselected three models with lowest prediction error computed by 10 folds cross-validation: (I) NLR + tobacco; (II) NLR + tobacco + Liang score; (III) respiratory impairment. Their performances are displayed and compared in Table 9. Surprisingly, LDH did not appear to predict death reliably while it was the central variable for ICU admission. The only factor that seems to provide reliable information regarding death is respiratory impairment. Indeed, a logistic regression based on this variable outperforms Liang score but its specificity remains low (24%). The two other considered models [i.e., (NLR + tobacco) and (NLR + tobacco + Liang score)] give similar predictive results. Logistic regression coefficients of competing models are shown in Tables 10-13. Nevertheless, Delong test comparing the AUC of the four more accurate predictive models for death does not show any statistically significant difference between them (Table 14). Therefore, we cannot pick and choose one model over another. The main information that can be drawn from this analysis is that the Respiratory impairment score shows heterogeneity in its components (Table 13). If the % LI is significant, neither the DLI nor the SpO₂ have significant coefficient. These findings are in line with the literature (30) and our clinical experience: SpO₂ is not a good criterion nor for death (Table 13) nor for admission to ICU (Table 7).

Table 9 Predictive performances of competing models regarding death
Full table

Table 10 Liang score logistic regression coefficients (death)
Full table

Table 11 (NLR + tobacco) score logistic regression coefficients (death)
Full table

Table 12 (Liang + NLR + tobacco) score logistic regression coefficients (death)
Full table

Table 13 Respiratory impairment score logistic regression coefficients (death)
Full table

Table 14 Delong test comparing AUC of the four more relevant predictive models for death
Full table

Discussion

Significant information can be drawn from this study. Firstly, a promising score from a large-scale study in China appears to perform poorly when applied to a European cohort, mainly by lack of specificity, whether to predict for admission to ICU or death. This finding suggests that genetic factor drive the “cytokine storm” in COVID-19 patients (31). Secondly, biological features that are quite significant for the admission to ICU such as LDH or venous lactate cannot predict death, suggesting that these two outcomes should be clearly distinguished. Thirdly, simple and affordable variables such as LDH, LDH + sex, or LDH + sex + venous lactate have a very good sensitivity and an acceptable specificity for ICU admission. From the logistic regression coefficients, it appears that a serum LDH value above 579 U/L and a serum lactate value greater than 3 mmol/L are good predictors for admission to the ICU. In addition, the odds ratio for gender-related risk for ICU admission is 1.71 higher for men than for women. This is quite in line with our experience where men fill about two-thirds of ICU beds. For death prediction, we report in our cohort a significant NLR cut-off of 22.1 and an aggravating role for tobacco consumption. However, these promising results are flawed by a lack of specificity. Fourthly, respiratory impairment is a promising model to predict death. Among the variables making up this score, the % LI is significant but not the DLI nor the SpO₂. These results confirm clinical-based findings: one should not rely on ambient SpO₂ to estimate respiratory distress at admission in the emergency ward. However, all these scores suffer from a lack of specificity and cannot be used reliably.

The main lesson from our research is that predictive scores from institutions outside our population catchment area, even if they have been calculated on a large scale, must be considered with great care. Genetic factors or selection bias may explain the differences in results. The major limitation of our study is of course the small number of patients enrolled.

A predictive score for admission to ICU or death is urgently needed in secondary hospitals such as Clinique Saint-Pierre Ottignies. Interleukine-6 (IL-6) has been shown as a marker of severity of the disease. Meta-analysis of mean IL-6 concentrations demonstrated three-fold higher levels in patients with complicated COVID-19 compared with patients with non-complicated disease (32). However, most developed countries hospitals cannot afford costly laboratory exams. Optimal allocation of resources guided by clinical-based indicators will best guide patients at time of admission and avoid futile treatments in ICUs. These indicators are still lacking. Wynants et al. have shown that proposed models in the literature are at high risk of bias and their reported performances probably optimistic (4). We show that a predictive score based on a large-scale population study may not be reliable enough to be implemented in Belgium, both for admission to the ICU and for death, mainly because of lack of specificity. However, in our small cohort it appears that LDH above 579 UI/L and venous lactate above 3.02 mmol/L may be considered as good predictive biological factors for ICU admission. On the other side, death risk can be assessed by NLR above 22.1, tobacco abuse status and respiratory impairment. The development of reliable predictive methods are of great importance to ensure fair allocation of scarce medical resources in time of COVID-19 (33). However, our results suggest that available methods may not have to require accuracy to be beneficial in emergency ward.

Acknowledgments

We warmly thank the patients who took part in the study and their advisors/families, as well as the entire nursing staff of the Clinique Saint-Pierre Ottignies.

Funding: The study was funded solely by departmental funding.

Footnote

Reporting Checklist: The authors have completed the MDAR reporting checklist. Available at http://dx.doi.org/10.21037/jeccm-20-173

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/jeccm-20-173). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with Declaration of Helsinki (as revised in 2013). The study was approved by the Institutional Ethics Committee of our hospital “Ethics Committee Clinique Saint-Pierre Ottignies” (Ethical Approval ID 2020-13). The euract number of the study is B0432021000000. Individual consent for this retrospective analysis was waived.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. COVID-19 Map. Johns Hopkins Coronavirus Resource Center. [cited 16 May 2020]. Available online: https://coronavirus.jhu.edu/map.html
2. Rubin GD, Ryerson CJ, Haramati LB, et al. The role of chest imaging in patient management during the COVID-19 pandemic: a multinational consensus statement from the Fleischner Society. Chest 2020;158:106-16. [Crossref] [PubMed]
3. Ye Z, Zhang Y, Wang Y, et al. Chest CT manifestations of new coronavirus disease 2019 (COVID-19): a pictorial review. Eur Radiol 2020;30:4381-9. [Crossref] [PubMed]
4. Wynants L, Van Calster B, Collins GS, et al. Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal. BMJ 2020;369:m1328. Update in BMJ 2021;372: Erratum in: BMJ 2020;369:m2204.
5. Collins GS, Reitsma JB, Altman DG, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): The TRIPOD Statement. Ann Intern Med 2015;162:55. [Crossref] [PubMed]
6. Liang W, Liang H, Ou L, et al. Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19. JAMA Intern Med 2020;180:1081-9. [Crossref] [PubMed]
7. Skedgel C, Wailoo A, Akehurst R. Societal preferences for distributive justice in the allocation of health care resources: a latent class discrete choice experiment. Med Decis Making 2015;35:94-105. [Crossref] [PubMed]
8. Moons KGM, Altman DG, Reitsma JB, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 2015;162:W1-73. [Crossref] [PubMed]
9. van Smeden M, Moons KG, de Groot JA, et al. Sample size for binary logistic prediction models: beyond events per variable criteria. Stat Methods Med Res 2019;28:2455-74. [Crossref] [PubMed]
10. Vittinghoff E, McCulloch CE. Relaxing the rule of ten events per variable in logistic and cox regression. Am J Epidemiol 2007;165:710-8. [Crossref] [PubMed]
11. Hastie T, Tibshirani R, Friedman JH. The elements of statistical learning: data mining, inference, and prediction: with 200 full-color illustrations. New York: Springer, 2001:533.
12. Molinari R, Bakalli G, Guerrier S, et al. SWAG: A Wrapper Method for Sparse Learning. Swiss Finance Institute Research Paper 2020. doi: 10.2139/ssrn.3633843.10.2139/ssrn.3633843
13. Pan F, Yang L, Li Y, et al. Factors associated with death outcome in patients with severe coronavirus disease-19 (COVID-19): a case-control study. Int J Med Sci 2020;17:1281-92. [Crossref] [PubMed]
14. Zheng Z, Peng F, Xu B, et al. Risk factors of critical & mortal COVID-19 cases: a systematic literature review and meta-analysis. J Infect 2020;81:e16-25. [Crossref] [PubMed]
15. Shipe ME, Deppen SA, Farjah F, et al. Developing prediction models for clinical use using logistic regression: an overview. J Thorac Dis 2019;11:S574-84. [Crossref] [PubMed]
16. Tolles J, Meurer WJ. Logistic regression: relating patient characteristics to outcomes. JAMA 2016;316:533-4. [Crossref] [PubMed]
17. Yuki K, Fujiogi M, Koutsogiannaki S. COVID-19 pathophysiology: a review. Clin Immunol 2020;215:108427 [Crossref] [PubMed]
18. Pons S, Fodil S, Azoulay E. The vascular endothelium: the cornerstone of organ dysfunction in severe SARS-CoV-2 infection. Crit Care 2020;24:353. [Crossref] [PubMed]
19. Bikdeli B, Madhavan MV, Jimenez D, et al. COVID-19 and thrombotic or thromboembolic disease: implications for prevention, antithrombotic therapy, and follow-up: JACC state-of-the-art review. J Am Coll Cardiol 2020;75:2950-2973. [Crossref] [PubMed]
20. Li X, Ma X. Acute respiratory failure in COVID-19: is it "typical" ARDS? Crit Care 2020;24:198. [Crossref] [PubMed]
21. Cheng Y, Luo R, Wang K, et al. Kidney disease is associated with in-hospital death of patients with COVID-19. Kidney Int 2020;97:829-38. [Crossref] [PubMed]
22. Bhatraju PK, Ghassemieh BJ, Nichols M, et al. Covid-19 in critically ill patients in the Seattle region - case series. N Engl J Med 2020;382:2012-22. [Crossref] [PubMed]
23. Yoshida T, Grieco DL, Brochard L, et al. Patient self-inflicted lung injury and positive end-expiratory pressure for safe spontaneous breathing. Curr Opin Crit Care 2020;26:59-65. [Crossref] [PubMed]
24. Kotfis K, Williams Roberson S, Wilson JE, et al. COVID-19: ICU delirium management during SARS-CoV-2 pandemic. Crit Care 2020;24:176. [Crossref] [PubMed]
25. Metlay JP, Waterer GW, Long AC, et al. Diagnosis and treatment of adults with community-acquired pneumonia. An official clinical practice guideline of the American Thoracic Society and Infectious Diseases Society of America. Am J Respir Crit Care Med 2019;200:e45-67. [Crossref] [PubMed]
26. Akhmerov A, Marbán E. COVID-19 and the heart. Circ Res 2020;126:1443-55. [Crossref] [PubMed]
27. Perico L, Benigni A, Casiraghi F, et al. Immunity, endothelial injury and complement-induced coagulopathy in COVID-19. Nat Rev Nephrol 2021;17:46-64. [Crossref] [PubMed]
28. Ackermann M, Verleden SE, Kuehnel M, et al. Pulmonary vascular endothelialitis, thrombosis, and angiogenesis in Covid-19. N Engl J Med 2020;383:120-8. [Crossref] [PubMed]
29. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988;44:837-45. [Crossref] [PubMed]
30. Shenoy N, Luchtel R, Gulani P. Considerations for target oxygen saturation in COVID-19 patients: are we under-shooting? BMC Med 2020;18:260. [Crossref] [PubMed]
31. Ye Q, Wang B, Mao J. The pathogenesis and treatment of the ‘Cytokine Storm’ in COVID-19. J Infect 2020;80:607-13. [Crossref] [PubMed]
32. Coomes EA, Haghbayan H. Interleukin-6 in Covid-19: a systematic review and meta-analysis. Rev Med Virol 2020;30:1-9. [Crossref] [PubMed]
33. Emanuel EJ, Persad G, Upshur R, et al. Fair allocation of scarce medical resources in the time of Covid-19. N Engl J Med 2020;382:2049-55. [Crossref] [PubMed]