Evaluation of a hypothetical decision-support tool for intensive care triage of patients with coronavirus disease 2019 (COVID-19) [version 1; peer review: awaiting peer review]

Background: At the start of the coronavirus disease 2019 (COVID-19) pandemic there was widespread concern about potentially overwhelming demand for intensive care and the need for intensive care unit (ICU) triage. In March 2020, a draft United Kingdom (UK) guideline proposed a decision-support tool (DST). We sought to evaluate the accuracy of the tool in patients with COVID-19. Methods: We retrospectively identified patients in two groups: referred and not referred to intensive care in a single UK national health service (NHS) trust in April 2020. Age, Clinical Frailty Scale score (CFS), and co-morbidities were collected from patients’ records and recorded, along with ceilings of treatment and outcome. We compared the DST, CFS, and age alone as predictors of mortality, and treatment ceiling decisions. Results: In total, 151 patients were included in the analysis, with 75 in the ICU and 76 in the non-ICU-reviewed groups. Age, clinical frailty and DST score were each associated with increased mortality and higher likelihood of treatment limitation (p-values all <.001). A DST cutoff score of >8 had 65% (95% confidence interval (CI) 51%-79%) sensitivity and 63% (95% CI 54%-72%) specificity for predicting mortality. It had a sensitivity of 80% (70%-88%) and specificity of 96% (95% CI 90%-100%) for predicting treatment limitation. The DST was more discriminative than age alone (p<0.001), and potentially more discriminative than CFS (p=0.08) for predicting treatment ceiling decisions. Conclusions: During the first wave of the COVID-19 pandemic, in a hospital without severe resource limitations, a hypothetical decision support tool was limited in its predictive value for mortality, but Open Peer Review Reviewer Status AWAITING PEER REVIEW Any reports and responses or comments on the article can be found at the end of the article. Page 1 of 11 Wellcome Open Research 2021, 6:183 Last updated: 22 JUL 2021


Introduction
In the first phase of the coronavirus disease 2019 (COVID- 19) pandemic, in March 2020, there was widespread concern in the United Kingdom (UK) that there would be insufficient intensive care unit (ICU) beds and mechanical ventilators to treat the number of patients presenting with severe COVID-19 1,2 . Pandemic modelling suggested that rates of infection in the UK would exceed ICU capacity 3 .
Even outside of a pandemic, not all patients with severe illness are admitted to intensive care. There are patients too well to benefit from ICU management, and others for whom the short or long-term benefit is uncertain or likely to be small relative to the burden of treatment 4,5 . In the event of there being intense pressure on ICU capacity, as was feared to be the case during the first wave of COVID-19, there could be a role for a system to support allocation of these resources in a fair and clinically effective way 6 .
The National Institute for Health and Care Excellence (NICE) published a rapid clinical guideline on critical care for adults in the context of COVID-19 on 20 th March, 2020 7 . It recommended that all admissions be assessed for frailty, using the Clinical Frailty Scale (CFS) 8 . It suggested that patients who are less frail "(for example a CFS score of less than 5)" would likely benefit from intensive care while for more frail patients (e.g. CFS 5 or above) there was uncertainty about benefit 7 . A draft UK national pandemic allocation guideline, developed by the UK Moral and Ethical Advisory Group (MEAG) in conjunction with the Intensive Care Society in late March 2020, proposed a scoring system incorporating age, frailty and co-morbidities ( Figure 1) 9,10 . This suggested benefit of intubation and ventilation for patients with a Decision Support Tool (DST) score of eight or below, while for patients with a DST of >8 it would be appropriate to limit treatment (potentially including continuous positive airway pressure (CPAP)/non-invasive ventilation). The MEAG guideline and tool was apparently rejected by UK health officials 11 , and no official NHS guidance was produced.
There were two potential justifications for thresholds proposed in the NICE and MEAG guidance. First, that high scores would be predictive of little benefit from treatment in patients with COVID-19 (i.e. low survival). This would then provide a basis for rationing treatment in the event of a critical shortage in the availability of intensive care 10 . Secondly, such scores might correlate with the decisions about the appropriateness of intensive care ordinarily made by ICU specialists. The scoring system would therefore allow rapid and consistent decisions by other members of the healthcare team. However, at the time that these were proposed, there were no available data on how well the CFS or DST thresholds would predict mortality from COVID-19, nor whether they would reliably predict decisions about ceilings of treatment.
The aim of this study was to evaluate the proposed MEAG decision-support tool (DST) and CFS threshold for predicting mortality and for identifying patients judged clinically appropriate for intensive care admission, in patients hospitalised with COVID-19 in a hospital without severe resource limitations during the first wave of the pandemic.

Methods
We sought to retrospectively identify all adult patients (aged 17 years and above) with COVID-19 referred in our trust for potential ICU admission in April 2020. Oxford University Hospitals NHS Foundation Trust (OUH) includes one tertiary centre, one specialist hospital and one district general hospital; there are a total of 1465 beds with a baseline of 60 intensive care beds. Medical record numbers of patients reviewed by ICU consultants were collected prospectively.
During this period, decisions about admission to intensive care for patients referred to ITU were made by the on-call ITU consultant on the basis of the patient's wishes, clinical condition and the perceived appropriateness of intensive care admission. There were no policies setting thresholds for admission, and the DST was neither used nor recorded. Patients receiving only non-invasive ventilation were largely treated outside on intensive care.
We retrospectively identified a control group of patients with COVID-19 admitted to OUH but not referred to intensive care. (Decisions about treatment for this cohort were made by the treating consultant/senior registrar on the basis of the patient's wishes and clinical appropriateness). Because of the larger number of patients, we restricted the control group to the two-week window from 8 th April to 21 st April 2020 to arrive at a cohort of the same size as the ITU group. Control patients were identified from patients identified on the trust electronic patient record system. Patients who did not have COVID-19 or 'COVID-syndrome' (presumed COVID-19 on the basis of clinical and radiographic findings but tested negative on swabbing) or did not a have an oxygen requirement were excluded.
The electronic patient records were examined by two independent data collectors (EST and BP). Disagreements were discussed and resolved by consensus or review by a third researcher (DW). Age, clinical frailty score, and specific co-morbidities identified as part of the DST were recorded (cardiac arrest in last 3 years from any cause, any chronic condition causing three or more hospital admissions in the last year or four or more weeks continuous admission for current inpatients, congestive heart failure or chronic lung disease with symptoms at rest or on minimal exertion, hypertension, severe and irreversible neurological condition, chronic liver disease with Child-Pugh score of ≥7, end stage chronic renal failure requiring renal replacement therapy, diabetes mellitus requiring medication, uncontrolled or active malignancy) 9 . Age was classified into subgroups as specified in the DST. At the time of the study, clinical frailty was recorded routinely for many admissions with COVID-19 as per NICE guidance. Where a frailty score was not documented, relevant information from the patient's electronic records, including from the initial clerking and reviews by physiotherapists and occupational therapists, was used to estimate this (based on consensus between data collectors). A patient who was a marathon runner at the time of admission would for example be assigned a score of 1, a patient who was reliant on support for all of their personal care but stable would have been assigned a score of 7, etc, per the NICE CFS.
Admission and review dates, whether the patient had a treatment escalation plan documented, treatment offered, and ceiling of treatment decisions during admission were recorded. Ceilings of treatment were classified into three categories designating the maximum level of support that would be potentially appropriate to provide: (1) intubation and ventilation, (2) non-invasive ventilation including high flow nasal oxygen and CPAP, and (3) administration of oxygen by other means (e.g. nasal cannula). If ceiling of treatment was not clearly documented, patients were excluded for analysis of this (but were still able to be included in mortality figures).
Hospital outcomes were recorded from electronic records. These were whether, as of the end of May 2020, the patient had been discharged, was still an inpatient, or had died. This project was registered as an audit with the Trust quality improvement team on 17 th April 2020. Ethics review was not required.
We compared the DST, CFS, and age alone as predictors of mortality, and treatment ceiling decision (i.e. ceiling of treatment not including intubation/ventilation). We assessed each predictor's performance using receiver operating characteristic (ROC) curves and the area under the curve (AUC). We also estimated each predictor's sensitivity and specificity at specific thresholds: for DST score a score of ≤8 vs >8; for the CFS, a score of ≤4 vs. >4; and for age, ≤75 vs. >75 years old.
For statistical analysis we used the statistical programming language R version 4.0.2 and the package pROC 12 . We computed sensitivity and specificity values for ROC curves using that package's roc function. To generate confidence intervals (CIs) for sensitivity and specificity of each predictor, we used the package's bootstrap intervals 13 . To generate confidence intervals for AUC values, we used pROC's bootstrap-based method; to test the pairwise differences between AUC values, we used the package's implementation of De Long's test 14 .

Results
In the month of April 2020, 749 patients over the age of 17 years tested positive for COVID-19 in OUH. A further 1085 had suspected COVID-19 without a laboratory test or with a laboratory test reported as inconclusive i.e. based on symptoms, radiological evidence, other blood test results. In April 2020, 70 patients with COVID-19 were treated in intensive care units in OUH, of which 57 were discharged and 13 died (18.6% mortality) 15 .
In total, 151 patients were included in this study, with 75 in the ICU and 76 in the non ICU-reviewed groups ( Table 1).
Ceiling of care was not clearly documented in 4 cases (2 ICU reviewed, 2 non-reviewed).
Patients who were reviewed by intensive care were younger, less frail, and had lower decision-support tool scores; Mann-Whitney tests produced negligible p-values (<.001). (Table 1, Figure 2). There were 4 patients reviewed by ICU with a CFS score <³5 and 9 with a DST score >8.

Decisions and outcome
Among patients referred for potential ICU admission, 57 (78%) had no treatment limitation documented, (27 of this 57 (47%) were intubated), while 16 (22%) had a ceiling of treatment of non-invasive ventilation documented. In the non ICU-reviewed group, 41 (55%) had a ceiling of treatment of nasal canulae or face mask oxygen delivery, 20 (27%) of non-invasive  ventilation (5/20 received NIV), and 13 (18%) of intubation and ventilation, should that need arise. Seven (9%) were treated in high dependency areas, none were intubated.

Prediction of mortality
Increased age, clinical frailty and DST score were each associated with increased mortality (separate logistic regressions of mortality on CFS, DST score, and age category index all produced p-values <.001) ( Table 2).

Prediction of clinical decision
Increased age, clinical frailty and DST score were each associated with a higher likelihood of a treatment limitation decision (p-values from separate logistic regressions of treatment decision on CFS, DST score, and age category index all <.001).
Of patients with a DST equal to or below 8, 15% (12/79) had a treatment limitation decision. Overall, 96% (65/68) of patients with a DST of more than 8 had a treatment limitation decision ( Table 2). A DST cut-off of more than 8 had a sensitivity of 80% (70%-88%) and a specificity of 96% (95% CI 90%-100%) for predicting treatment limitation.

Discussion
In this single centre study, among patients hospitalised with COVID-19 in a UK hospital during the first wave of the pandemic, a hypothetical triage scoring tool (DST) had moderate sensitivity and specificity for predicting in-hospital mortality (65% and 63%). The tool appeared to be highly predictive of clinical decisions about the appropriateness of intubation and ventilation, made in the absence of severe resource shortage. The DST (which includes components of age, comorbidities and clinical frailty) appeared to be more discriminatory than age or clinical frailty alone.
When the study was first conceived, it was unclear whether a national guideline in relation to intensive care triage would be forthcoming. It was felt to be important, if possible, to evaluate the accuracy of such a tool before it was adopted. On 25 th October 2020 an official statement was issued by the NHS in response to allegations that had been made that patients who were frail and elderly had been denied adequate care during the first wave of the COVID-19 pandemic. The statement emphasised that the NHS had not run out of critical care beds in this time period, and that intensive care was offered to the patients who were assessed and deemed likely to benefit from it on an individual basis. It acknowledged that a tool had been developed to facilitate triaging of patients, but that this was neither formally launched nor was use of the NHS logo on it authorised.
Previous pandemic plans have focused on illness severity scores, sometimes in combination with exclusion of patients with severe life-limiting comorbidity (e.g. metastatic malignant disease). For example, the Ontario Health Plan for an Influenza epidemic (OHPIP) was recommended for use in a future influenza pandemic 16 . It combines Sequential Organ Age (as a discrete factor) p = 6.6 × 10 -4 (χ 2 = 23.4 on 6 degrees of freedom)

Table 3. Mortality and treatment limitation per clinical frailty (CFS) and Decision Support Tool (DST) scores.
ICU=intensive care unit.
Clinical frailty has been proposed as a tool for intensive care triage because of its relationship with outcome for critically ill older patients. A 2017 meta-analysis found that frailty was associated with higher in-hospital and long-term mortality 20 .
In the "VIP1" (Very old Intensive Care Patients) study, 5021 patients aged 80 years or over were followed up after admission to intensive care units in 21 European countries 21 . There was a linear relationship between the clinical frailty score and mortality. The authors observed about a 30% 30-day mortality for patients with a CFS of 3 requiring emergency intensive care admission, rising to 75% mortality for CFS 9. However, the absolute rate of mortality in older frail patients admitted to intensive care varies between studies [22][23][24][25] .
While the use of clinical frailty in COVID-19 triage has been defended 10 , there are other factors that are relevant to decisions about the benefit and appropriateness of intensive care. Co-morbidity has been clearly linked to mortality in patients with COVID-19, and is a risk factor for length of stay in intensive care 26,27 . Age also appears to be an independent risk factor for intensive care outcome 26,27 . Younger patients who are clinically frail also have increased intensive care mortality. In a Canadian cohort of about 200 patients aged 50-65 years, frailty was linked with increased mortality risk over ensuing 12 months in adjusted analysis 28 . The authors concluded that the probability of survival is higher for younger compared with older frail patients, but their risk is age-shifted compared to peers -i.e. they have a mortality risk equivalent to an older chronological age.
There are some limitations to our study. The surge in Oxford and the United Kingdom as a whole was significantly less than anticipated and the hospitals' increased capacity through reorganisation of services was sufficient to offer intubation and ventilation to patients using the same or very similar standards as under non-COVID-19 circumstances. Intensive care and medical doctors were thus not faced with decisions to ration care in the way that was predicted. The numbers of patients in this single centre study are relatively small, and triage scores were evaluated retrospectively from medical records, which may have introduced inaccuracy or bias in classification. A small proportion did not have clear documentation of ceiling of treatment. The decision-making and outcomes reported in this study may be different from those in other centres with greater pressure on intensive care, or indeed from those in our own centre at other time points during the pandemic. The mortality rate was lower than average for the UK (overall, in the first wave there were 10,624 UK ICU admissions with COVID-19, of which 6232 were discharged, 4109 died -a mortality rate of 38.7%) 29 .
Since the clinical frailty score was included in NICE guidance relating to COVID-19 management, it is possible that knowledge of CFS directly influenced decisions about treatment and therefore the calculated sensitivity/specificity may not be accurate. (However, in half of the cohort, the CFS was not documented in the medical record, potentially meaning that at least in that group it did not directly influence decisions).
Since treatment limitation decisions may lead to higher mortality, the relationship between the DST and mortality is potentially influenced by self-fulfilling prophecies 30 . However, in our cohort the DST did not appear to be more predictive of mortality than either frailty or age alone. Calculated values for sensitivity and specificity obtained from a case-control cohort may be different from an unselected cohort. A larger prospective multi-centre study from a multi-organisational study would have generated more accurate assessment. Hewitt et al. found greater correlation between CFS score and mortality in their multi-centre study 31 . Since this decision-support tool was proposed, other prognostic risk stratification scores for patients with COVID-19 have been assessed in large cohorts e.g. ISARIC (International Severe Acute Respiratory and Emerging Infection Consortium) 32 .

Conclusions
To our knowledge, this is the first study to formally evaluate the potential performance of the proposed MEAG decision support tool. While the DST was not used, it was a strong predictor of clinical decisions made by the doctors in the OUH hospitals; in our patient cohort it appeared to have better discriminatory power than age alone, and potentially better power than the clinical frailty score. When it comes to predicting mortality, however, the three metrics we considered performed similarly. This data may be of use in the assessment of other triage tools in this or future pandemics.