Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) antibody lateral flow assay for antibody prevalence studies following vaccination: a diagnostic accuracy study

Background: Lateral flow immunoassays (LFIAs) are able to achieve affordable, large scale antibody testing and provide rapid results without the support of central laboratories. As part of the development of the REACT programme extensive evaluation of LFIA performance was undertaken with individuals following natural infection. Here we assess the performance of the selected LFIA to detect antibody responses in individuals who have received at least one dose of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) vaccine. Methods: This was a prospective diagnostic accuracy study. Sampling was carried out at renal outpatient clinic and healthcare worker testing sites at Imperial College London NHS Trust. Two cohorts of patients were recruited; the first was a cohort of 108 renal transplant patients attending clinic following two doses of SARS-CoV-2 vaccine, the second cohort comprised 40 healthcare workers attending for first SARS-CoV-2 vaccination and subsequent follow up. During the participants visit, finger-prick blood samples were analysed on LFIA device, while paired venous sampling was sent for serological assessment of antibodies to the spike protein (anti-S) antibodies. Anti-S IgG was detected using the Abbott Architect SARS-CoV-2 IgG Quant II CMIA. A total of 186 paired samples were collected. The accuracy of Fortress LFIA in detecting IgG antibodies to SARS-CoV-2 compared to anti-spike protein detection on Abbott Assay Results: The LFIA had an estimated sensitivity of 92.0% (114/124; 95% confidence interval [CI] 85.7% to 96.1%) and specificity of 93.6% (58/62; 95% CI 84.3% to 98.2%) using the Abbott assay as reference standard (using the threshold for positivity of 7.10 BAU/ml) Conclusions: Fortress LFIA performs well in the detection of antibody responses for intended purpose of population level surveillance but does not meet criteria for individual testing.

patients attending clinic following two doses of SARS-CoV-2 vaccine,

Introduction
As vaccination programmes for coronavirus disease 2019 (COVID-19) are rolled out worldwide, population antibody testing is useful in monitoring immune responses to vaccinations, informing discussion and decisions about booster doses, and assessing levels of potential protective immunity in the population 1 .
Lateral flow immunoassays (LFIAs) have the potential to deliver affordable, large-scale testing of individuals and provide rapid results without the support of central laboratories Antigen lateral flow testing is already being used widely. This approach, using antibody lateral flow devices, has been used across England in the REACT2 (REal time Assessment of Community Transmission) 2 study to estimate the number of infections during the first wave of the COVID-19 pandemic 2 , monitor the decline in antibody positivity over time 3 and assess population antibody prevalence following vaccine roll-out, most recently in Round 5 of the study published in February 2021 4 .
Prior to the scale up of antibody testing for surveillance, extensive clinical and laboratory evaluation of diagnostic accuracy following natural infection was performed on a range of LFIA antibody tests 5,6 , identifying one for subsequent use. The test selected (Fortress, Northern Ireland) detects antibody against the spike protein of the virus (contained in all licensed vaccines) and would therefore be expected to detect vaccine induced antibody responses. This study examined the accuracy of the Fortress LFIA device in detecting antibodies in two cohorts of vaccinated individuals and explored the relationship between LFIA results and viral neutralisation.

Methods
This was a prospective diagnostic accuracy study conducted between 20th December 2020 and 26 th May 2021. Samples were collected from two groups: renal transplant patients (cohort 1) and healthcare workers (cohort 2).

Bias
Every attempt was made to address potential sources of bias. All eligible participants were offered enrolment where practical and every effort was made to ensure understanding of the participant information sheet (PIS) and study procedure, using translation services where necessary. Potential participants were given time to consider participation and trained research staff were able to answer questions relating to the study.

Eligibility criteria
Eligibility for both cohorts was defined as: 2) Able to understand and consent to study 3) Received either one or two doses of any UK approved vaccine for COVID-19 4) Able to comply with study procedure/ study not thought to be risk to patient

Sample size
Sample size was computed based on an expected sensitivity of 90% and specificity of 95%, with a minimal acceptable lower confidence limit of -10% for both estimates. Under power 1 -β = 0.85 and α = 0.05, the minimum sample size required is 106 cases and 76 controls. Patients were pragmatically enrolled to ensure minimum sample size achieved.  Table 1 for full details). All patients provided written informed consent.
Lateral flow immunoassay testing. Participants were supplied with an LFIA testing kit as used in the REACT home testing programme 7 . The LFIA (Fortress, NI) detects IgG and IgM

Amendments from Version 1
This updated manuscript acknowledges the comments made by our peer reviewers.
As a result of the pragmatic study design, some participants in the HCW cohort (13) have provided two samples to the study, at different time points (>21 days apart), which have been treated as independent samples for the purpose of testing the accuracy of the LFIA devices against gold-standard. We agree that if this study was looking at individual differences in participants, it would not be appropriate to include both sets of samples as this would introduce bias. In individuals who have provided two samples, it is likely that both the quantitative and qualitative immune responses will be different. As such given that the primary outcome for this study is to test the diagnostic accuracy of the LFIA device against the gold-standard Abbott titre we feel it is helpful to include these samples in the analysis. Acknowledging the reviewers comment, we have included a section addressing this in the discussion section and have shown the proposed sensitivity analysis removing the duplicate participants samples' in the supplement.
We have included a discussion of the confidence intervals and included reference for reported diagnostic accuracy of the assay for natural infection. The power has been edited to correct a previous error. Table 2 has been edited to correct a previous error. Figure 1 has a Y axis label added.

REVISED
to the S1 subunit of the spike protein. Participants were also provided with verbal instructions on how to use the test by a member of the research team, prior to performing self-testing, with support provided where necessary. The LFIA result was assessed independently by two observers. The results were reported by the colour intensity of the IgG band, and documented as either a positive or negative result. (cohort 2), quantitative antibody titres were reported in AU/ml. To allow combination with cohort 1 data, these were converted to BAU/ml by multiplying by 0.142. Double antigen binding assay (DABA) testing for discordant results (positive LFIA with negative serological) was performed on available stored samples from cohort 1. Detailed methodology of DABA has been described previously. Briefly, the Imperial Hybrid DABA is a sequential two step double binding assay for the detection and measurement of antibody directed to the receptor binding domain of SARS-CoV-2. The proteins employed were expressed and gifted by the Crick Institute, London. In order to evaluate specificity the Hybrid DABA was tested on stored plasma and serum samples predating the SARS-CoV-2 outbreak (n=825) in which 0 samples tested positive, giving a specificity of 100%.
In addition, for cohort 2, individuals provided blood for assessment of neutralisation assays. The ability of sera to neutralise the SARS-CoV-2 virus was assessed by neutralisation assay on Vero cells. Sera were serially diluted in OptiPRO SFM (Life Technologies) and incubated for 1h at room temperature with 100 TCID50/well of SARS-CoV-2/England/IC19/2020 and transferred to 96-well plates pre-seeded with Vero-E6 cells. Serum dilutions were performed in duplicate. Plates were incubated at 37°C, 5% CO 2 for 42 h before fixing cells in 4% PFA (paraformaldehyde). Cells were treated with methanol 0.6% H 2 O 2 and stained for 1h with a 1:3000 dilution of 40143-R019 rabbit mAb to SARS-CoV-2 nucleocapsid protein (Sino Biological). A 1:3000 dilution of sheep anti-rabbit HRP (horseradish peroxidase) conjugate (Sigma) was then added for 1 h. TMB (3,3′,5,5′-Tetramethylbenzidine) substrate (Europa Bioproducts) was added and developed for 20 mins before stopping the reaction with 1M hydrogen chloride (HCl). Plates were read at 450nm and 620nm and the concentration of serum needed to reduce virus signal by 50% was calculated to give NT50 values.

Performance analysis
The primary outcome of the study was sensitivity and specificity of the LFIA device in detecting SARS-CoV-2 IgG antibodies identified by the Abbott platform.
A secondary analysis was conducted using reference standard as either Abbott or, for discordant results (positive LFIA negative serology) in cohort 1 using in house DABA as reference standard for serological positivity.
Outcomes are presented with the corresponding binomial exact 95% confidence interval (95% CI

Cohort characteristics
The characteristics of the participants are described in Table 1 and Table 2. In total, 186 samples were tested using both LFIA and serological testing 9 .

LFIA IgG positivity and antibody titres in serum
The combined results describe both cohort 1 and cohort 2 (n=186 samples, Figure 1). Of those samples which scored positive on LFIA (n=118), 4 had undetectable laboratory anti-S levels using Abbott Architect assay. Three of these samples (from the renal transplant cohort) were subsequently re-tested using an in-house DABA which detected antibodies in 1 sample and confirmed negativity in 2. The remaining 114 samples had a median anti-S titre of 229.5 BAU/ml and mean of 229.5 BAU/ml; anti-S titre ranged from 9.7 BAU/ml to 5680 BAU/ml. Of those which scored negative on LFIA (n=68), anti-S antibodies were detected in 10 samples, of which 7 had anti-S titre levels <10 BAU/ml (7.  Figure 2).

Live virus neutralisation
Neutralisation titres were available for 64/78 samples in the healthcare worker cohort. Neutralisation titres (NT50) were significantly higher in those with positive LFIA compared to those without (Figure 3a). Only one LFIA-negative sample had detectable neutralisation assay using a threshold for positivity of (NT50 of 15 with an anti-S antibody titre of 7.8 BAU/ml).
For individuals with detectable IgG on LFIA only 2/34 did not have significant evidence of viral neutralisation.

Discussion
This study demonstrates that the Fortress LFIA device performs well in detecting IgG antibodies in vaccinated individuals when comparing against a serological assay widely used in routine practice. LFIAs have been a helpful tool the assessment of population antibody prevalence of SARS-CoV-2, and can play a role in informing vaccination strategy going forwards. The Fortress LFIA has been assessed previously for its performance following natural infection 6 , though did not meet Medicines and Healthcare products Regulatory Agency (MHRA) criteria for individual use which recommend antibody tests should have a sensitivity of >98% (95% CI 96% to 100%) and specificity of >98% on a minimum of 200 known negative controls 10 . The test has undergone extensive evaluation for home self-testing 7 and has since been used widely in community studies of antibody prevalence in England.
The performance of the LFIA in the cohorts of vaccinated individuals here demonstrates slightly higher sensitivity than previously reported for natural infection, though this difference is not significant. This is likely to reflect higher background titres of antibody following vaccination, particularly after second doses, when compared to natural infection in the community, at least in the healthcare worker cohort. The LFIA device does not detect very low levels of antibody which may still correlate with protection from severe disease and/ or hospitalisation. However, in the general population, the number of such individuals with low titres following two vaccinations will be low (in contrast to the renal transplant cohort studied here).
A small number of LFIA tests appear to produce false positive results (n=4) with undetectable antibodies in the commercial laboratory assay. To understand whether these were genuine false positives, these four samples were tested with a second sensitive assay (DABA). Only one if these discordant samples tested positive.
There is growing evidence that the presence of neutralising antibodies in sera is highly predictive of protection from symptomatic COVID-19 disease 11,12 . Although the LFIA studied has a threshold below which it can't detect Spike specific antibody that is present, that threshold is close to the level  at which neutralising antibody can be reliably measured (Figure 3a). This suggests that antibody positivity on the LFIA may give some indication of protection from symptomatic disease and thus could be useful to measure any waning of vaccine induced immunity in different populations.
The study has some limitations. The LFIAs were self-tested in the clinic or vaccination centre, where participants had access to support from trained healthcare professionals when required. This study does not fully replicate the 'real-world' application of LFIAs where users will be following a detailed guide in their own homes. Furthermore, the patient cohort includes healthcare workers and as such may have greater understanding and/or experience of self-testing than members of the general population. For this reason (and due to the relatively small sample size of this study with wide confidence intervals) there is a place for further studies with larger sample sizes in the community.
As a result of this pragmatic study design, some participants (13) in the HCW cohort provided two samples, at different time points, which were analysed within the study. As it is likely that there are differences in both the quantitative and qualitative immune responses in these samples, we analysed these samples independently given the primary purpose of the study was to evaluate the diagnostic accuracy of the LFIA compared to gold. If the 13 participants who provided two samples were removed from the calculations, the estimated sensitivity of the LFIA would be 87.7% with a wider (95%-CI 75.8-97.1) for HCW cohort and to 91% (95%-CI 84.1-95.6) overall. The performance of the LFIA evaluated is sufficiently good that it can continue to play a helpful role in the assessment of population antibody responses resulting from widespread infection and high levels vaccination coverage, particularly given the correlation of LFIA results with the functional measure of live virus neutralisation. Over time, antibody titres will begin to wane and ongoing population surveillance can play a helpful role in informing decisions on policy for subsequent vaccination programmes, the targeting of booster vaccines. Rapid antibody testing may prove useful in initial screening of patients to receive monoclonal antibody therapy as lab methods may cause a delay in therapy to potentially eligible patients.

Hans-Michael Kaltenbach
Department of Biosystems Science and Engineering, Swiss Institute of Bioinformatics (SIB), ETH Zürich, Basel, Switzerland This study determines the diagnostic accuracy of the Fortress LFIA for SARS-CoV-2 IgG antibodies after vaccination against an established reference assay. Specificity and sensitivity are estimated from samples of two cohorts and pooled estimates are reported as the main result. The authors provide an in-depth analysis of false-positive results and report additional results of neutralisation assays for one cohort. The results show lower sensitivity and higher specificity compared to the two cited previous evaluations of this LFIA, but both differences are not statistically significant.
Overall, the study is well documented and the availability of raw data allows easy reproduction of the results.
My only concern is the sample design and analysis of the HCW cohort. The calculations assume that the 78 samples are independent, but they are in fact pairs from 39 individuals measured before and after vaccination. I would sympathize with neglecting this issue for cases with a negative result at day 0 followed by a positive result at day 21, which one might treat as 'sufficiently' independent; however, one is not surprised to find the 13 positive samples of day 0 to also test positive at day 21. An ad-hoc remedy for this double-counting is to remove 13 of these 26 samples from the calculation as they provide no new information and artificially inflate the sample size. My only concern is the sample design and analysis of the HCW cohort. The calculations assume that the 78 samples are independent, but they are in fact pairs from 39 individuals measured before and after vaccination. I would sympathize with neglecting this issue for cases with a negative result at day 0 followed by a positive result at day 21, which one might treat as 'sufficiently' independent; however, one is not surprised to find the 13 positive samples of day 0 to also test positive at day 21. An ad-hoc remedy for this double-counting is to remove 13 of these 26 samples from the calculation as they provide no new information and artificially inflate the sample size.