Development and implementation of a customised rapid syndromic diagnostic test for severe pneumonia

Background: The diagnosis of pneumonia has been hampered by a reliance on bacterial cultures which take several days to return a result, and are frequently negative. In critically ill patients this leads to the use of empiric, broad-spectrum antimicrobials and compromises good antimicrobial stewardship. The objective of this study was to establish the performance of a syndromic molecular diagnostic approach, using a custom TaqMan array card (TAC) covering 52 respiratory pathogens, and assess its impact on antimicrobial prescribing. Methods: The TAC was validated against a retrospective multi-centre cohort of broncho-alveolar lavage samples. The TAC was assessed prospectively in patients undergoing investigation for suspected pneumonia, with a comparator cohort formed of patients investigated when the TAC laboratory team were unavailable. Co-primary outcomes were sensitivity compared to conventional microbiology and, for the prospective study, time to result. Metagenomic sequencing was performed to validate findings in prospective samples. Antibiotic free days (AFD) were compared between the study cohort and comparator group. Results: 128 stored samples were tested, with sensitivity of 97% (95% confidence interval (CI) 88-100%). Prospectively, 95 patients were tested by TAC, with 71 forming the comparator group. TAC returned results 51 hours (interquartile range 41-69 hours) faster than culture and with sensitivity of 92% (95% CI 83-98%) compared to conventional microbiology. 94% of organisms identified by sequencing were detected by TAC. There was a significant difference in the distribution of AFDs with more AFDs in the TAC group (p=0.02). TAC group were more likely to experience antimicrobial de-escalation (odds ratio 2.9 (95%1.5-5.5)). Conclusions: Implementation of a syndromic molecular diagnostic approach to pneumonia led to faster results, with high sensitivity and impact on antibiotic prescribing.


Introduction
For many decades the diagnosis of infectious diseases has relied on a combination of clinical assessment and microbiological culture. However, cultures are frequently negative 1,2 and can take several days to return a result 3 . Optimising antimicrobial therapy can be challenging, especially in patients who are at risk of multidrug resistant organisms 2 . In critically ill patients, this frequently results in the empiric use of broad-spectrum agents, with predictable consequences for antimicrobial resistance and other forms of antimicrobial-related harm such as drug toxicity and disruption of the microbiome 4 . Conversely, failure to identify the causative organism can lead to inappropriate antimicrobial therapy, which is associated with poor outcomes 5 .
Pneumonia amongst intubated and mechanically ventilated, critically ill patients can be especially difficult to diagnose 6 . Most critically ill patients are systemically inflamed 7 , clinical examination is unreliable 8 and there are multiple causes of radiographic lung infiltrates, most of which are non-infectious 9,10 .
The development of host-based biomarkers for infection, such as C-reactive protein 11 , procalcitonin 12 , and alveolar cytokine concentrations 10,13 have been advanced as useful measures to help rationalise antimicrobial use. However, their utility in the diagnosis 11,12 and antimicrobial stewardship 14,15 of pneumonia has been challenged.
There is, therefore, a pressing need for rapid, sensitive, multipathogen-focussed diagnostic tests for pneumonia 16 . However, although intensive care physicians appreciate the potential advantages of such diagnostics, they are also wary of potential downsides 17 , emphasising the need for evaluation of these tests in real-life clinical practice.
TaqMan array cards (TAC) enable the conduct of multiple simultaneous single-plex real-time polymerase chain reaction (RT-PCR), with this format allowing rapid and straightforward customisation. This customisation allows for a wider range of organisms than those found in existing commercially available tests, and rapid modification to address emerging threats. Although TACs have shown promising performance relative to conventional microbiology 18 , our previous experience demonstrated that a TAC with restricted coverage of common respiratory pathogens had a limited impact on clinical decision making in critically ill patients 19 . We therefore set out to develop and implement a multi-pathogen array that would have broad applicability for severe pneumonia.

Ethical and regulatory approvals and funding
The prospective study was approved by the Leeds East Research Ethics Committee (17/YH/0286), Cambridge University Hospitals NHS Foundation Trust was the sponsor, and registered with clinicaltrials.gov (NCT03996330). The assessment of routinely collected data from the comparator group received a consent waiver as data came from routinely collected clinical data and was conducted under a protocol approved by the institutional review board (A095506). The protocol has been deposited on Zenodo 20 . VAPrapid 15 was approved by the England and Northern Ireland (13/LO/065) and Scotland (13/SS/0074) National Research Ethics Service committees and sponsored by Newcastle upon Tyne Hospitals NHS Foundation Trust.

Card development
The local microbial ecology was reviewed using previous conventional microbiological culture data from the hospital. This was supplemented by review of the literature concerning causative organisms reported in ventilator-associated and community-acquired pneumonia and the authors' previous experience of molecular diagnostics in pneumonia 1,2,6,13,19,21 . Species-or genus-specific primer/probe sequences were identified by reviewing the literature for well cited and fully validated real-time PCR assays with the presumption that, where possible, each organism should be covered by two sequences to minimise false positive results. In the absence of a published validated assay, one was designed in-house, normally targeting a housekeeping gene in the first instance (i.e. gyrB, rpoB, ssrA, dnaJ, recN) following the guidelines set out previously 22 . Briefly, all assays were subjected to a comprehensive in silico analysis using BLAST analysis to ensure specificity of primers and probes and examined for possible cross-reactions with other high priority organisms. Potential for adverse probe/ primer interactions and melting temperature (T m ) were assessed using OligoAnalyzer 3.1. If necessary, sequences were modified accordingly to remove any cross reactions and to ensure T m of 55-60°C, to allow for uniform amplification. The 52 organisms (23 bacteria, 2 mycobacteria, 6 atypical bacteria, 5 fungi and 16 viruses) covered by the card are shown in Figure 1. Sequences on the card have been deposited on Zenodo 23 .

TAC procedure
Nucleic acid extraction from clinical samples was undertaken using the NUCLISENS easyMAG platform (cat number 280140, Biomerieux, Marcy L'Etoile, France), in accordance with the manufacturer's instructions. Nucleic acids were extracted from 500 µL of input sample, with a dilution of MS2 bacteriophage added pre-extraction to act as an internal extraction and inhibition control.
Cards were run on the QuantStudio 7 Flex platform (cat number 4485701,ThermoFisher, Waltham, MA, USA), following a modified version of the method previously described 18 . Briefly, 50 µL of each nucleic acid extract was mixed with

Amendments from Version 2
Following the comments from the second reviewer we have added a paragraph to the discussion concerning the changes in the pulmonary microbiome which can occur during mechanical ventilation in critical illness and how this can be challenging to distinguish from true infection, by both conventional and molecular microbial diagnostics.
Any further responses from the reviewers can be found at the end of the article   (Table 1).

Retrospective cohort validation.
A retrospective cohort validation was conducted using stored bronchoalveolar lavage (BAL) samples obtained during the 24 centre VAPrapid trial of a biomarker for the diagnosis of ventilator-associated pneumonia 15 . VAPrapid centres used semi-quantitative microbiological culture as the reference standard but did not undertake routine testing for viruses or Pneumocystis jirovecii. The stored samples were analysed on the TAC using methods outlined above.
Prospective evaluation. The protocol for the prospective study, finalised prior to first recruitment, has been deposited on Zenodo 20 . The prospective study was registered with clinicaltrials.gov (NCT03996330; June 24, 2019).

Setting
Patients were recruited from a 20-bedded teaching hospital Intensive Care Unit (ICU). The unit is a mixed general medical-surgical unit which supports transplant and haematology-oncology services.

Recruitment
Between February 2018 and August 2019, patients were eligible for inclusion if they were receiving invasive mechanical ventilation, and if the treating intensive care specialist (consultant) suspected pneumonia and was planning to perform diagnostic bronchoscopy. Exclusions were lack of a proxy decision maker to provide study assent, and lack of laboratory study team availability to perform the TaqMan array card assay (the laboratory study team were routinely unavailable from Friday 5pm to Monday 8am, and also sporadically unavailable due to leave). Patients were included consecutively when the study team was available. Written informed consent was obtained when patients had capacity at time of enrolment. For those lacking capacity, written proxy assent (nominated or personal consultee advice) was obtained prior to study inclusion, and retrospective consent was sought if capacity was regained whilst the patient remained in hospital. Patients were identified by the treating team and included prospectively and consecutively when the laboratory study team were available.
Patients who were not included in the study because of a lack of TAC laboratory team availability, and those from the month prior and month following the study, formed the comparator group.
Sampling procedure Bronchoscopy for both TAC and comparator patients was conducted in accordance with the unit protocol. Briefly, the scope was wedged in a sub-segment of the area with maximal radiographic change, or where frank pus was seen emerging. In diffuse infiltrates a sub-segment of the right middle lobe or lingula was selected. Saline in 50ml aliquots up to a volume of 200ml was instilled and withdrawn after a dwell time of 10 seconds. Where samples were taken out of hours (Monday-Thursday 5pm-8am and Sunday 8am-Monday 8am), samples were stored at 4°C prior to processing within 24 hours, in accordance with existing laboratory procedures.

TAC testing
The samples for TAC were processed as set out above. The TAC was run by a dedicated laboratory team who did not undertake the conventional PCR or cultures, with blinding also assured by the results of the TAC being obtained before those from conventional microbiology. As an experimental assay, the results of the TAC were not included in the laboratory information system, blinding the assessors of the reference standard to the TAC results.
Return of results to clinical team Following review by a consultant clinical scientist, results were returned to the ICU team. Clinical microbiology advice was available 24 hours/day, and patients underwent weekday daily combined ICU-Microbiology multi-disciplinary reviews in keeping with existing unit practice (weekend microbiology input was available on request). The study did not mandate any course of action by the treating clinical team. Conventional microbiology results were returned to clinicians via the electronic health record; however, in practice these were returned after the TAC results.

Outcome measures
The co-primary outcome measures were sensitivity, using conventional microbiology as the reference standard and time to result compared to conventional microbial culture. Time to result for microbial culture was taken as time from completion of lavage to first organism identification, or confirmation of negative growth if no organisms were detected.
Secondary outcome measures were sensitivity compared to metagenomic microbial sequencing, time to result compared to conventional PCR, days alive and free of antibiotics (antibiotic-free days, AFDs) in seven and 28 days following lavage and change in antibiotic therapy in the seven days following lavage. Qualitative assessment of whether TAC results impacted on antimicrobial change was assessed by clinical notes review by a member of the study team who was not involved in the decision-making process (VW).

Statistical analysis
The difference in median time to result for conventional culture and TAC was assessed by Wilcoxon's matched-pairs test.
Where conventional PCR failed, or where the lab did not test for the organism, the corresponding tests from the TAC were removed from calculation of diagnostic performance. Indeterminate cultures ('mixed upper respiratory tract flora') were considered negative. A sensitivity analysis, coding failed conventional PCR and organisms not tested 'negative' was also undertaken. Comparisons of distribution of antibiotic free days between TAC and comparator groups was by Mann-Whitney U test, differences in proportions of escalation and de-escalation decisions were assessed by Chi 2 test. Analyses were conducted using Prism v9.1 (Graphpad Inc, La Jolla, CA).
Study size A planned prospective study size of 100 patients evaluated by TAC was selected to balance cost against including sufficient numbers to be able to make a judgement on the card's clinical utility. As the co-primary endpoint was time to result in a realworld setting that had not been previously evaluated, we did not undertake a formal power calculation.

Metagenomic sequencing
To further validate the results of the prospective TAC assay, metagenomic sequencing was undertaken. Residual BAL samples (average 40 mL) from 98 out of the 100 patients were used for metagenomic sequencing (two could not be sequenced due to presence of potential containment level 3 organisms and lack of a CL3 facility in the sequencing laboratory). BAL was centrifuged at 500 x g for five minutes to separate the host cells (pellet) from the bacterial, viral and fungal pathogens (supernatant). One mL of each sample supernatant was filtered through a 0.45 µm filter and used for viral RNA and DNA extraction using a QIAamp MinElute Virus Spin kit (Qiagen), using an on-column DNase step for viral RNA. Reverse transcription and random amplification of both viral DNA and cDNA was carried out as described previously 25 . The remaining supernatant was centrifuged at 3220 x g for 30 minutes and the pellet was subjected to host cell depletion using MolYsis Basic5 (Molzym, Bremen, DE) followed by bacterial/fungal DNA extraction using a QIAamp DNA Mini kit (Qiagen, Hilden, DE). Half of the DNA was submitted for HiSeq 4000 shotgun metagenomic sequencing, while the other half was used to amplify the 16S V4 region using barcoded primers 26 (The Earth Microbiome Project), amplicons were sequenced by Illumina MiSeq sequencing. All samples were sequenced at the Wellcome Sanger Institute, and raw read data are available at the European Nucleotide Archive (ENA) project PRJEB29011 with study accession numbers ERP111277, ERP111280, ERP112277, and ERP018622. Amplicon data were analysed using Qiime2 v2019.10.0 27 . Single-end sequences were denoised using Deblur 28 , and classified using a feature classifier built from the Greengenes 13_8 99% OTUs taxonomy database. For metagenomic shotgun sequence data, human reads were first removed from the data using Bowtie2 v2.3.5 29 . 5'Humandepleted paired reads were then classified using Kraken2 v2.0.8 30 . For bacterial targets, a curated bacterial database based on the Genome Taxonomy Database 31 was used for classification. For viral and fungal pathogens, the standard Kraken2 viral and fungal databases were used. Qiime2 and Kraken2 tabular outputs were subsequently processed in R (version 3.5.3) (R core team) to calculate the proportions of reads mapping to individual taxa for each sample.
The outputs for each of the sequencing approaches were compared to paired negative controls and analysed for the presence of fungal, viral and bacterial reads. Fungal and viral organisms were reported if they were the dominant species and/or the read counts were above the determined background levels.
Bacterial organisms that were reported if identified by both shotgun and 16S amplicon sequencing, or present as the dominant species in shotgun sequencing above the background levels. Organisms that had low read counts, or were only identified by 16S but were detected by TAC, were reported as low confidence hits.

Technical validation
Following initial validation against stored DNA extracts and synthetic plasmids, all microorganisms from the Quality Control for Molecular Diagnostics 2018 Sepsis EQA Pilot Study were successfully detected (Table 1).

Retrospective cohort validation
The card was tested against the stored samples available from the VAPrapid study 15 . 128 samples with semi-quantitative culture results were available for analysis. 57 organisms were grown at or above 10 4 colony forming units (CFU)/ml 24,32 , with 55 detected by TAC ( Table 2). The TAC detected a further 295 organisms, including 64 viruses and one atypical organism which the recruiting centres did not test for. Excluding tests for organisms not detectable by culture, 3425 tests on TAC were negative. Sensitivity was 97% (95% confidence interval (CI) 88-100%) and specificity 94% (95% CI 93-95%) ( Table 3). Organisms detected by both TAC and culture had a median cycles to threshold (Ct) value on the TAC of 29 (interquartile range (IQR) 26-32 range 20-35) whilst culturable organisms detected on TAC but not on culture had a median Ct value of 33 (IQR 30-35 range 20-40) ( Figure 2).

Prospective evaluation
Between January 2018 and September 2019, 166 ventilated patients were investigated for pneumonia by bronchoscopy, and 95 were tested by TAC. No proxy decision maker approached refused consent, and 24 patients who regained capacity whilst in hospital were approached and all gave retrospective consent. Five patients were tested twice by TAC, having suffered a subsequent respiratory deterioration, so in total 100 TACs were run. 71 patients formed the comparator group ( Figure 3). Although inclusion criteria were pragmatic and only required senior clinician suspicion of pneumonia, 92% of cases met full ECDC criteria for clinical pneumonia (Figure 4). Of the eight cases not meeting full ECDC criteria, one lacked a formal radiological report of infiltrates, one had no clinical signs of pneumonia, five had no signs of systemic inflammation and one patient lacked both radiological and systemic inflammation. Table 4 shows participant characteristics of the study population and comparator group.
Time to result. The median difference in time to result between TAC and conventional culture was 51 hours (IQR 41-69 hours p<0.0001 by Wilcoxon matched pairs), the TAC also returned results more rapidly than conventional PCR in almost all cases ( Figure 5). The minimum TAC time to return was 4 hours, with median time to result 22 hours (IQR 7-24 hours), most of the delays arose from samples taken outside routine working hours, whilst additional delays with conventional PCR results largely reflect laboratory workflow and batching of samples.

Comparison of organisms detected by TAC compared to conventional microbiology.
178 organisms were identified from 100 samples on the TAC (Table 5). Details of individual sample results are available as extended data 33 . Conventional microbiology detected 66 organisms, with 61 detected by TAC. 27 patients had failure of internal control for one or more conventional PCR assays, covering 93 organisms. There were no TAC internal control failures and none of the organisms covered by the failed assays were detected on TAC or sequencing (Table 5). Sensitivity and specificity were 92% (95% CI 83-98%) and 97% (95% CI 97-98%) respectively (Table 6). Including failed and absent reference standards as 'negative' had minimal effect on diagnostic performance (Table 7).
Comparison by sequencing. 98 samples were available for sequencing. Metagenomic sequencing revealed 107 organisms, 100 of which were also detected by TAC (Table 5 and extended data Table 1 33 ).
Concerning the 10 organisms detected by conventional microbiology or sequencing but missed by TAC, one organism, that was positive by both culture and sequencing albeit in different patients, was Citrobacter freundii, for which we did not have a sequence on the card. A further five pathogens were detected by sequencing (Staphylococcus aureus, Legionella spp., and Staphylococcus epidermidis) or both culture and sequencing (two E. faecium). Although these five were detected by TAC, they did not pass the internal quality control standards required for reporting and were considered 'negative' results. The remaining three organisms, two rhinovirus by conventional PCR and one Staphylococcus spp. by sequencing, were not detected by TAC at all.
One case of Aspergillus fumigatus was detected on the TAC, and although no moulds were cultured, the lavage galactomannan antigen test was highly positive (5.92 optical density index (ODI), laboratory reference range <0.5 ODI).
Quantitation. Twenty-five organisms were grown on conventional culture at ≥10 4 CFU/ml, the conventional cut off for quantitative culture of lavage 24,32 . The median Ct for these organisms on the TAC was 27 (IQR 24-29, range 20-33). In contrast, culturable organisms detected on TAC but not on culture, and therefore likely to be present in lower concentrations and not reported for patients managed without TAC, had a median Ct of 32 (IQR 30-34, range 22-38) (Figure 2).
Antibiotic prescribing. Patients in the TAC and comparator cohorts had similar severity of illness, severity of respiratory failure and demographic features (Table 4). Patients managed with the TAC had a significantly different distribution of AFDs to the comparator group in the seven days following bronchoscopy (p=0.02 by Mann-Whitney U-test), with more AFDs in the TAC cohort ( Figure 6). This difference did not retain significance over 28 days (Figure 7). Overall 72 (76%) of TAC patients had their antibiotics changed in the seven days following bronchoscopy, with a total of 116 changes made (Table 8). In the comparator group 50 (70%) of patients experienced a total of 65 changes. Whilst 63% of decisions in the TAC group led to de-escalation, only 37% of decisions in the comparator group were de-escalation decisions (OR 2.9 (95% CI 1.5-5.5) p=0.008 by Chi-squared). Decisions which were judged to be related to the TAC result were weighted further towards de-escalation (73% of all TACrelated changes, Table 8). 11 (30%) of escalations in the TAC group were judged to have been targeted escalations in response to TAC results. In a further six cases negative TAC results prompted investigation for alternative diagnoses.

Discussion
We demonstrate that a customised molecular diagnostic, designed to meet the needs of a specific clinical setting, produced accurate results in a clinically important time-frame and was associated with an increase in antibiotic-free days relative to the comparator group in the week following investigation. Diagnostic performance was similar when assessed in stored samples from multiple centres, implying a generalisable result.
Molecular diagnostic platforms for respiratory infection syndromes have, until recently, largely focussed on viral pathogens 16 .
However, the need to optimise antimicrobial therapy whilst limiting the over-use of these drugs has led to repeated calls for bacterial-focussed diagnostics 16,34 . TACs have been previously reported for use in pneumonia 19,35,36 . However, apart from our previous report 19 that demonstrated limited clinical impact due to restricted organism coverage, none of the other reports have included ventilated patients and were restricted to retrospective analysis of stored samples. Commercial multiple-pathogen arrays that include respiratory bacteria have recently become available, however most reports of their use in ventilated patients remain limited to describing diagnostic performance, and reporting 'potential' to change antimicrobial therapy rather than impact on clinical practice 3,37,38 . Concerns have been raised about the risks of over-treatment from molecular diagnostics 16,17,39,40 , whilst conversely promising tests with the potential to change therapy have not always proven this in clinical practice 15,19 . These commercially available assays lack the broad coverage and customisability of the TAC, with consistent concerns raised around limited organism coverage adversely impacting treatment decisions 3,17,19,37,38 .
There is now widespread acceptance of the presence of a respiratory microbiome 41,42 and the lungs of ventilated patients present a challenge to highly sensitive molecular diagnostics 16,17 .
The proximal respiratory tract of ventilated patients becomes rapidly colonised with predominantly Gram negative organisms 43,44 . This can occur in the absence of infection, and there is a risk that highly sensitive techniques will detect colonising organisms, driving unintended increases in antimicrobial use 16 . The use of protected lower airway specimens, with growth ≥10 4 CFU/ml for BAL have been used to distinguish infection from colonisation 22,32,45 . We adapted this approach in this study, using the quasi-quantitative Ct value provided by RT-PCR and testing protected bronchoalveolar samples. Using the comparison of the Ct values of organisms detected by culture and those detected by TAC without culture, we suggest that a Ct threshold of 32 be used to suggest infecting rather than colonising organism ( Figure 2 and Figure 8).
One of the problems that has beset bacterial diagnostics studies has been the absence of a 'gold standard' against which the candidate can be assessed 16,34,46 , as conventional culture is imperfect. For this study we supplemented comparison against conventional microbiology (culture and viral PCR) with metagenomic sequencing. 10 organisms identified by conventional microbiology or sequencing were not detected by the TAC. Overall, the TAC detected more organisms than either culture or sequencing, reflecting the higher sensitivity of qPCR. However, without a perfect validation method we cannot be certain these were not 'false positives' and have counted them as such for the calculation of specificity. The sequencing and culture results give clinicians considerable confidence in the results provided.
The selection of organisms targeted on the card was crucial, and informed by our previous experience where omission of key organisms significantly limited the impact of a similar card 19 . Given the case mix of our unit, with a high proportion of immunosuppressed patients, we opted to include a number of low pathogenicity organisms, (i.e. coagulase-negative Staphylococci (CNS), Enterococci and Candida albicans), as well as Herpesviridae, which were routinely tested for prior to this study. The detection of these organisms can be challenging to interpret 47 , given that many critically ill patients have a degree of immunoparesis, even if not classically immunosuppressed 6,48 , and so their significance remains uncertain. As our laboratory routinely reported these organisms on conventional microbiology the clinical team were already confronted with this issue.
It is increasingly apparent that the microbiome of the lungs of ventilated patients undergoes a shift, with increasing dominance of enteric organisms which may enter the lungs via gut Determining whether enteric organisms, even those such as E coli which are conventionally considered pathogenic in the lung, are the cause or the consequence of pulmonary inflammation remains a challenge. However this is a challenge which besets both conventional and molecular diagnostics, and one which is likely to take further developments in host-response diagnostics to resolve 10,13,15,41,44 The inclusion of CNS aids with the interpretation of the detection of the mecA gene, which is commonly carried by these organisms, thus helping identify MRSA. The lack of CNS sequences on commercial cards has been noted to impair interpretation of mecA detection on other molecular diagnostic platforms 37,51 . The ready customisability of the TAC would allow units to remove such organisms, as well as add other organisms that emerge as a threat as we have done subsequently during the COVID-19 pandemic 52 . Although we undertook the steps noted in the methods section to validate the TAC prior to use, the range of organisms encountered in the retrospective and prospective cohorts' clinical samples did not cover the full range of organisms on the TAC. Some organisms not seen in this report had been detected in previous studies using the same assays in an earlier iteration of the card 18,19 . We must therefore be somewhat cautious about the interpretation of the assays without clinical sample validation, and continue to monitor card performance as experience with this card develops in our centre.
The use of a contemporaneous comparator cohort allowed for comparisons of antibiotic prescribing within the context of the implementation of the TAC and any heightened awareness of antimicrobial stewardship it may have engendered. Despite this, the comparator cohort saw a greater proportion of escalation decisions in the week following lavage, and had   Totals 66 5134 5200 Figure 6. Distribution of days alive and free of antibiotics in the seven days following bronchoscopy and lavage in the TAC and comparator cohorts. Following first lavage only for patients who had more than one bronchoalveolar lavage during ICU admission. Numbers in each category and percentage shown below graph, p value by Mann-Whitney U test.
fewer antibiotic-free days. The 28 day mortality was similar between the two groups, and both had similar duration of ventilation and ICU length of stay (Table 4), which suggests that the change in antibiotic therapy was not associated with harm. The lack of difference in AFDs at day 28 is unsurprising, as suspected pneumonia is only one of multiple drivers of antibiotic use. Although the comparator and TAC groups had similar characteristics, our observational design means that we cannot be certain that unmeasured confounders did not contribute to the effects seen. Replication in additional settings with distinct approaches to stewardship is required before we can be certain of its external generalisability, whilst evaluation in a randomised, controlled trial would help reduce any bias that may have arisen from our observational study design.

Implementation of TaqMan array
Our experience leads us to suggest the following approach to using the TAC. Where a pathogenic organism(s) is detected at Ct value of ≤32, antimicrobials can be adjusted to target the organism(s) detected, in light of known local resistance patterns and the patient's history carriage of antimicrobial-resistant organisms. Pathogenic organisms detected at a Ct of >32 are likely to be colonisers or contaminants, although detection of respiratory viruses or obligate pathogens such as Legionella spp or Mycoplasma pneumoniae at higher Ct values remains significant. The detection of low pathogenicity organisms needs to be interpreted in light of the patient's known or suspected immune status. Among the immunocompromised, high levels of such organisms, especially if it is the sole pathogen detected, may prompt treatment. In patients in whom no relevant pathogens are detected (i.e. all organisms are at low levels, low pathogenicity organisms are detected in immunocompetent hosts, or no organisms are detected at all) consideration should be given to alternative sites of infection, alternative diagnoses and where clinical suspicion of infection is low, stopping antibiotics ( Figure 8). We used the TAC in the context of an existing daily microbiology-ICU multi-disciplinary team meeting, and as noted above generalisability of our findings to other contexts will require replication in those settings. We Table 8. Detail of changes in antibiotic therapy in the seven days following lavage in the TAC and comparator cohorts. Changes judged to be TAC-related are shown in the left-hand sub-column for the TAC group. Several patients had more than one change in antibiotic therapy. *includes two de-escalations to prophylactic dose, ** includes two escalations from prophylactic to therapeutic dose.

TAC-related TAC-unrelated
De-escalation Stopping macrolides 14 3 6 Stopping carbapenem or anti-pseudomonal penicillin 10 14 10 Narrowing from carbapenem/antipseudomonal penicillin to narrower spectrum penicillin note that further clinical sample validation is required for rare organisms not encountered in our retrospective and prospective cohorts. Confirmation of safety of TAC-driven changes in antimicrobials will be best established in prospective randomised trials.

Conclusions
This study established a molecular diagnostic test to meet the needs of a particular intensive care unit, although generalisability was demonstrated through testing samples from multiple other units. The strength of the TAC is that it allows customisation and rapid modification to address emerging threats and ensure broad coverage. Implementation in the context of an antimicrobial stewardship program led to significant impact on antimicrobial prescribing. We believe this approach represents a promising new approach to the management of severe pneumonia but this requires testing in the context of well-designed randomised clinical trials. Patient data is not publicly available for confidentiality reasons in line with the approved study protocol, but anonymised data can be obtained through contact with the corresponding author, subject to appropriate data sharing agreements being in place. Data sharing agreements will be arranged by Cambridge University Hospitals NHS Foundation Trust, as the study sponsor.
MM-resources, investigation, writing-review and editing.
TH-resources, investigation, writing-review and editing EH-investigation, writing-review and editing.
SF-investigation, writing-review and editing.
JD-investigation, writing-review and editing.
SP-investigation, writing-review and editing.
PP-investigation, data curation, writing-review and editing.
JB-investigation, data curation, writing-review and editing.
LT-investigation, data curation, writing-review and editing.
WS-data curation, writing-review and editing JS-resources, investigation, writing-review and editing AR-resources, investigation, writing-review and editing MR-data curation, writing-review and editing.
DS-data curation, writing-review and editing.
MET-conceptualisation, investigation, data curation writing-review and editing RMcM-resources, investigation, writing-review and editing DE-investigation, writing-review and editing.
VW-formal analysis, investigation, data curation writing-review and editing.
VAPrapid investigators-resources and investigation.

Open Peer Review
specimens in patients with pneumonia. They show that their approach yields highly sensitive results in significantly shorter time as compared to time required for culture-dependent assays. In addition, they demonstrate that the use of TAC led to a significant reduction in antibiotic free days, which could be of importance in improving antibiotic stewardship practices. Overall, the manuscript is well written and the methodological steps are sound. This manuscript has been reviewed extensively prior to my review, and I do not have much to add.
However, I agree with reviewer 1 that certain pitfalls will have to be addressed before such methods can be used as part of widespread clinical practice. A specific point of concern is the use of TAC on broncho-alveolar lavage samples leading to the detection of high amounts of bacterial communities originating from the gastrointestinal tract (i.e. Escherichia coli and Enterococcus spp.). It is unclear to what extent these bacteria contribute to clinical disease, as recent studies have shown that it is likely that these microbial communities translocate from the gut into the lower airways following the development of a respiratory infection. Therefore, identification of these bacteria complicates rational antibiotic selection, as it is unclear if these bacteria are the cause or the effect of the disease. This point has been raised by reviewer 1 in the context of the high detection rates of Candida in the TAC cohort. However, I would request the authors to write an additional paragraph in which these specific limitations are addressed.
Reviewer Expertise: Infectious disease, (lung) microbiome I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. Are the conclusions about the method and its performance adequately supported by the findings presented in the article? Yes Reviewer Expertise: Infection diagnostics, antibiotic pharmacology I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
Author Response 05 May 2022

Andrew Conway Morris,
We thank Dr Felton for his review and comments. To address each in turn R1: Commercial multiplex PCR diagnostic assays would be expected to demonstrate the sensitivity and specificity of each of the individual assays. Whilst this is onerous given that some of the pathogens are uncommon, some of the PCR targets are not present at all in the validation cohort.

A1:
We acknowledge that whilst the samples tested in both the retrospective cohort validation and prospective evaluation contained a wide range of organisms they did not include all organisms for which we have sequences on the card. Previous studies (Steensals D et al and Jones N et al, references 18 and 19 in the manuscript) used many of the same assays used in this TAC, and identified some organisms not seen in our cohort. As noted in the methods, 'technical validation section', we did test the sequences on the cards against synthetic control plasmids to ensure their technical function, and also tested against extracts from known positive cultures in the lab. For atypical organisms where we did not have positive cultures in the lab (C burnetti. C psittaci, C pneumonia and M pneumoniae) commercially available controls were purchased from Vircell, Granada, Spain. The only organism which was validated entirely by synthetic control plasmid was Elizabethkingia meningoseptica, which was a prevalent organism in our hospital 5 years ago but has not been detected on culture recently. We agree that detection in clinical samples with confirmed culture of these organisms would be the gold standard. With the card being used in Cambridge University Hospitals for adult pneumonia, and an ongoing study in paediatric pneumonia we have continued to accumulate data and review card performance regularly. Through this, some of the targets that were not validated in clinical samples this study have subsequently been validated. We have adjusted the methods to include further information regarding technical validation against clinical culture and commercial controls, and altered the discussion section to highlight this potential issue.
The specific changes are methods "…….covering all targets except C burnetti. C psittaci, C pneumonia, M pneumoniae and Elizabethkingia meningoseptica. For C burnetti. C psittaci, C pneumonia and M pneumoniae control DNA extracts were purchased (Vircell, Granada, Spain, Catalogue numbers MBC018, MBC013, MBC011 and MBC035 respectively). " Discussion "Although we undertook the steps noted in the methods section to validate the TAC prior to use, the range of organisms encountered in the retrospective and prospective cohorts' clinical samples did not cover the full range of organisms on the TAC. Some organisms not seen in this report had been detected in previous studies using the same assays in an earlier iteration of the card 18,19 . We must therefore be somewhat cautious about the interpretation of the assays without clinical sample validation, and continue to monitor card performance as experience with this card develops in our centre." R2:The authors make no comment of safety related to changing antibiotic prescribing. Given that some of the targets are unvalidated and the PCR result may have resulted in antibiotics being withheld or stopped early it would be helpful to understand the outcome of patients.
A2:We agree that safety is an important outcome, and note in the introduction that potential for harm may cut both ways (reference 4 summarising the evidence of antimicrobial harm, whilst reference 5 speaks to the risks of delayed and inappropriate therapy). Although we reported mortality being similar between the comparator and prospective TAC evaluation groups (table 4) in the context of demographically similar groups, we have now included data showing similar duration of ICU stay and duration of ventilation. We have added a note of these values in the discussion, and we noted previously that a randomised study would be the best way to address the potential of residual confounding.
R3:It is unclear to me how the TaqMan assay results in a reduction in antibiotics given the huge number of pathogens identified in patients. The authors state that the results were released into clinical practice without specific advice but presumably some advice was given as to which pathogens not to treat? Additionally the diagnostic accuracy analysis is a little misleading given how much better the new test performs compared to the gold standard. Is a conventional diagnostic accuracy analysis the best analysis to perform? A3: We examined the antibiotic therapy following lavage, and whilst we agree that the TAC did detect more organisms we also found that when changes occurred this tended to be towards de-escalation, although there were examples of targeted escalations as well. Although we have looked at decisions around the initiation or cessation of individual antibiotics and their relationship to the TAC, we acknowledge in the paper that this judgement is inherently subjective. This is why our primary analysis of antibiotic use is overall antibiotic-free days between the groups and change in antibiotics, which are taken from the drug prescriptions and thus do not require subjective judgement. This is likely to under-estimate the impact of the TAC as some antibiotics may be prescribed for reasons other than suspected pneumonia.
In terms of microbiology advice, being in the study did not mandate a specific course of action. Whilst results were released to the clinical team, our unit has a daily microbiology-ICU multi-disciplinary round where all microbiology results (including those from the TAC) are considered and decisions made around antimicrobial prescription. This is noted in the methods, 'return of results to the clinical team' section. In the discussion we noted "Replication in additional settings with distinct approaches to stewardship is required before we can be certain of its external generalisability". We have further emphasised this in the discussion section 'implementation of the TaqMan array' with an additional sentence "We used the TAC in the context of an existing daily microbiology-ICU multi-disciplinary team meeting, and as noted above generalisability of our findings to other contexts will require replication in those settings." We agree that conventional measures of diagnostic accuracy are difficult in evaluation of tests with greater sensitivity than the existing clinical standard. This does run the risk of understating the apparent specificity of the test, as we note in the discussion "Overall, the TAC detected more organisms than either culture or sequencing, reflecting the higher sensitivity of qPCR. However, without a perfect validation method we cannot be certain these were not 'false positives' and have counted them as such for the calculation of specificity" however we felt that understating the apparent specificity and acknowledging this risk was the best approach. Equally we acknowledge that culture may miss organisms that are not culturable, which is why we undertook the sequencing. The lack of a consensus standard on the interpretation of sequencing for clinical samples meant that we didn't include these data as part of the primary outcome measure of sensitivity (which was vs conventional microbiology), but reported them separately. However we think that the data from both conventional microbiology and sequencing are supportive of the diagnostic performance of the TAC. We feel that presenting the summary diagnostic test performance as % sensitivity and % specificity is the most easily interpretable presentation, and is consistent with our pre-defined study protocol outcomes.
R4:A lot of Candida species was identified in the respiratory tract. Candida is generally considered a coloniser in the lung and not a cause of pneumonia. Did the authors treat each of the cases of candida? More detail of the management of candida in the lung is requireddid these patients all receive antifungal therapy? If candida was regarded as a colonising organism it should be considered a negative result in the diagnostic accuracy analysis.

A4:
We agree that candida in the lung are not normally considered a cause of pneumonia and in only two cases were anti-fungals started by the treating clinicians on the basis of detection of candida (table 8), one of these patients was immunocompromised and the other had an oesophageal perforation and both were considered to be at risk of systemic candidiasis. The third initiation of antifungals noted in table 3 was in response to the detection of aspergillus on the TAC. Our diagnostic accuracy referred to the detection of organisms rather than the detection of pneumonia, so we compared all organisms detected on TAC to those detected by conventional microbiology without inferring a diagnosis of pneumonia. Pneumonia remains a difficult diagnosis to make in ventilated patients and we think that detection or non-detection of organisms is a more objective measure. We discuss our reasons for including 'low pathogenicity organisms' on the TAC in the discussion section.
R5:Overall a really interesting manuscript. I think a number of issues need to be addressed before this assay can be rolled out into clinical practice: validation of the rare pathogens, patient safety, out performance of the gold standard. It would strengthen the manuscript if the authors added a section to the discussion to outline next steps. This would make it clearer that the assay is not ready for mass adoption (which is suggested).

A5:
We thank Dr Felton for these comments. As per A1 above, we agree that assays for rare organisms that have not been previously validated need to be validated in clinical samples alongside our culture and synthetic plasmid validation. We also agree that testing at other sites, ideally in the context of a randomised controlled trial, would help address the questions of generalisability and safety of this approach. We have added further sentences to the discussion section, implementation of the TaqMan array and conclusions to emphasise these points. These additional sentences are "We note that further clinical sample validation is required for rare organisms not encountered in our retrospective and prospective cohorts. Confirmation of safety of TAC-driven changes in antimicrobials will be best established in prospective randomised trials" and "We believe this approach represents a promising new approach to the management of severe pneumonia but this requires testing in the context of well-designed randomised clinical trials."