Interpreting estimates of coronavirus disease 2019 (COVID-19) vaccine efficacy and effectiveness to inform simulation studies of vaccine impact: a systematic review [version 1; peer review: awaiting peer review]

Background: The multiple efficacious vaccines authorised for emergency use worldwide represent the first preventative intervention against coronavirus disease 2019 (COVID-19) that does not rely on social distancing measures. The speed at which data are emerging and the heterogeneities in study design, target populations, and implementation make it challenging to interpret and assess the likely impact of vaccine campaigns on local epidemics. We reviewed available vaccine efficacy and effectiveness studies to generate working estimates that can be used to parameterise simulation studies of vaccine impact. Methods: We searched MEDLINE, the World Health Organization’s Institutional Repository for Information Sharing, medRxiv, and vaccine manufacturer websites for studies that evaluated the emerging data on COVID-19 vaccine efficacy and effectiveness. Studies providing an estimate of the efficacy or effectiveness of a COVID-19 vaccine using disaggregated data against SARS-CoV-2 infection, symptomatic disease, severe disease, death, or transmission were included. We extracted information on study population, variants of concern (VOC), vaccine platform, dose schedule, study endpoints, and measures of impact. We applied an evidence synthesis approach to capture a range of plausible and consistent parameters for vaccine efficacy and effectiveness that can be used to inform and explore a variety of vaccination strategies as the COVID-19 pandemic evolves. Open Peer Review Reviewer Status AWAITING PEER REVIEW Any reports and responses or comments on the article can be found at the end of the article. Page 1 of 21 Wellcome Open Research 2021, 6:185 Last updated: 16 NOV 2021


Introduction
It took just over a year from when the World Health Organization declared the coronavirus disease 2019 (COVID- 19) epidemic to be a global pandemic for multiple vaccines to be authorised for emergency use 1,2 with many more in the pipeline 3 . The rapid development of vaccines using a variety of technology platforms (Table 1) has been remarkable and provides the first preventative mass intervention against COVID-19 that does not rely on economically damaging social distancing measures.
Randomised controlled trials (RCTs) of COVID-19 vaccines have demonstrated high efficacy against symptomatic infection, with a range of primary endpoint efficacy estimates reported between 95.0% (95% confidence interval (CI): 90.3 -97. 6) for Pfizer (BNT162b2) and 50.7% (95% CI: 36.0 -62.0) for CoronaVac. These studies have also demonstrated higher efficacy against severe outcomes (including hospitalisation and death) [4][5][6][7][8][9][10] . Whilst direct comparison of these estimates is not possible (due to differences in trial design, endpoint definitions of symptomatic disease, and the epidemiological setting in which the trials were undertaken), they can be useful to assess the likely impact of vaccine introductions on local epidemics. However, the vaccine efficacy measured in tightly controlled RCTs is not always observed under "real-world" conditions 11 . Thus, as vaccines have been rolled-out, real-world data on their effectiveness against symptomatic infection, severe disease, asymptomatic infection 12 and onward transmission 13,14 are emerging.
Synthesising these data is challenging given the speed with which data are emerging and the different study designs. Furthermore, global vaccine deployment means that target populations will be highly heterogeneous even at national level. Achieved coverage can vary by age, sex, ethnicity, comorbidities, and socio-economic background, all of which are known risk factors for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection and/or more severe COVID-19 disease outcomes 15,16 . Even within RCTs, COVID-19 vaccine efficacy has varied by human immunodeficiency virus (HIV) status 17 or dosing strategy 6 . Additionally, countries may adopt alternative roll-out strategies from those tested in RCTs. For example, the United Kingdom (UK) adopted a 12-week gap between vaccine doses based on early data suggesting that relatively high efficacy against symptomatic disease is conferred by the first dose in order to prioritise wider delivery of the first dose 18,19 and other countries such as Canada, Norway, and Denmark, are now taking a similar approach 20-22 . Conversely, due to delays in vaccine delivery for countries participating in the COVAX Facility, a longer gap between first and second doses may be inevitable 23 . Coupled with this variability, the emergence of multiple variants of concern (VOCs) that carry mutations in the spike protein 24 mean that the effectiveness of the current vaccines could be reduced (notably against B.1.351, P.1, and B.1.617.2 which were first identified in South Africa 25 , Brazil 26 , and India 27 , respectively). Whilst data on the vaccine efficacy against these VOCs remains limited, laboratory-based neutralisation assays are suggestive of a likely reduction for some, but not all, of the vaccines 28,29 .
As countries continue mass vaccination campaigns, policymakers must carefully consider the implications of different strategies. Optimal vaccination strategies are critically dependent on vaccine effectiveness, availability, roll out speed, and uptake. Relaxation of non-pharmaceutical interventions (NPIs) must be carefully balanced against progress in vaccination to prevent a surge in case numbers whilst population immunity remains below herd immunity threshold 30,31 . Simulation studies can provide useful insights into highly uncertain medium-term dynamics of SARS-CoV-2 transmission under different vaccine prioritisation and NPI lifting strategies, capturing inherent uncertainties in vaccine characteristics [34][35][36][37] .
Here we review the emerging data on COVID-19 vaccine efficacy and effectiveness, taking into consideration the study population, VOCs in circulation, and vaccine platforms. We take an evidence synthesis approach to capture a range of plausible and consistent parameters that can be used to inform and explore a variety of vaccination strategies. This article is reported inline with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines 38 .
Vaccine manufacturer websites were also searched for any press releases reporting COVID-19 vaccine efficacy or effectiveness. The search was supplemented by reviewing the reference list of relevant papers. Finally, all reports on vaccine effectiveness released on the UK government's website were also reviewed 41 . The review protocol was not registered at PROSPERO ahead of data extraction. All records were imported into the Covidence systematic review software and deduplicated (v2014, accessed 2021). Study title and abstracts were initially screened by one reviewer (NI) and full texts meeting the inclusion criteria were reviewed (NI).

Inclusion criteria
Studies providing an estimate of the efficacy or effectiveness of a COVID-19 vaccine using disaggregated data against SARS-CoV-2 infection, symptomatic disease, severe disease, death, or transmission were eligible for review. Studies assessing protection in a specific patient setting e.g. amongst patients with liver damage, were excluded. Press releases were included for review if results were not already available as a pre-print manuscript or peer-reviewed paper. Studies were grouped by unique vaccine for the evidence synthesis.

Data collection
A single reviewer (NI) extracted summary data on study type (clinical trial or effectiveness study), study location, vaccine type, efficacy or effectiveness by endpoint given as percentage vaccine efficacy, odds ratio, or risk ratio (infection, symptomatic disease, severe disease, death, or transmission), variant, number of doses, timing at which efficacy or effectiveness was assessed, and whether it was assessed in a particular subgroup.

Evidence synthesis
Although the primary endpoint of clinical trials is typically efficacy against symptomatic COVID-19 disease, optimising vaccination strategies requires consideration of broader transmission dynamics and the full natural history of disease from infection to recovery or death. We therefore consider vaccine efficacy 11 against four endpoints: i) any infection (including asymptomatic); ii) symptomatic infection; iii) severe disease requiring hospitalisation (including admission to intensive care units or resulting in death); and iv) infectiousness of breakthrough infections in vaccinated individuals.
Where only a single efficacy estimate was available, we use that value. Where several trials and/or effectiveness studies report values, we present the range of these values and use the unweighted mean value across studies as our central estimate. Efficacy or effectiveness estimates within care home populations were excluded from the calculation of the mean value as these study populations are highly vulnerable and experienced high attack rates early in the pandemic, making estimates non-representative. Efficacy estimates from non-standard dosing regimes e.g. the low/standard dosing reported in the ChAdOx1 trial were also excluded.
Based on the extracted values, we apply a data synthesis approach to generate working estimates. First, we assume that efficacy after two doses is never lower than after one dose for a multidose vaccine. Second, to ensure consistency between the endpoints we assume that efficacy against severe disease is greater than or equal to efficacy against symptomatic disease across all vaccine types [4][5][6] . Third, we assume that efficacy against B.1.351 and P.1, but not B.1.1.7 is never higher than that for wild-type SARS-CoV-2 (the variant originally in circulation that contains no major mutations) based on data on neutralising antibody levels 28,[42][43][44] . Estimates for VOCs with immune escape properties (B.1.351, P.1, and P.2) were grouped together. These three conditions were applied across all estimates.
Where data are not yet available, we first assume that the mechanism of action of the vaccines is similar. Specifically, to estimate vaccine efficacy or effectiveness (VE) against e.g. severe disease or infectiousness we maintain the same incremental relationships as observed for vaccines for which these data are available. For example, if data on efficacy against severe disease for B. We make a similar assumption for the difference in efficacy between first and second doses. If there was not enough information available to calculate these ratios, we assumed that: i) the odds of the vaccine efficacies for B.1.1.7 were the midpoint between those for wild-type and B.1.351; ii) that the efficacy for first and second dose was the same; or iii) efficacy against the same endpoint was the same for the different variants.
Second, in the absence of data, we assume that efficacies against symptomatic disease and infection are the same 45,46 . Third, for the AstraZeneca vaccine, immunogenicity data shows no significant boosting effect from the second dose 47,48 . Furthermore, efficacy after the first AstraZeneca dose against symptomatic disease was not significantly different to efficacy after the second dose in clinical trials 47 . Thus, in the absence of data we therefore assume that first and second dose efficacy for viral vector vaccines is the same. It is possible that the second dose may prolong the duration of vaccine-induced immunity, but we do not consider duration of protection in this analysis. Fourth, again in the absence of data, we assume that vaccines using the same vaccine platform had similar efficacies.
Additionally, in our analysis we impose a minimum and maximum vaccine efficacy against any endpoint of 0% and 99% respectively. We do not consider the duration of vaccine-induced immunity as data are not yet available to assess this. We did not undertake a risk of bias assessment as the aim of the review was to collate and synthesise all available efficacy and effectiveness estimates. All values are rounded to the nearest percent. Analysis was conducted using R version 4.0.3 49 and Microsoft Excel.

Results
We identified 602 potentially relevant studies or reports, 53 of which were included after full-text screening ( Figure 1).

Figure 1. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram of study selection.
The Underlying data file 50 summarises efficacy/effectiveness values reported from clinical trials and observational effectiveness studies. Figure 2 and Table 2 give our summary estimates of vaccine effectiveness by endpoint, vaccine type, dose, and SARS-CoV-2 variant. There were 16 studies reporting clinical trial results, 36 effectiveness studies, and 1 observational study.

AstraZeneca-Oxford (viral vector)
We identified 12 eligible studies on vaccine efficacy or effectiveness for the AstraZeneca vaccine ( Figure 3) which directly informed 8 out of 24 possible estimates in our evidence synthesis (Table 2. We estimated a 64%, 76%, and 99%, efficacy after two doses for wild-type against any infection, symptomatic disease, and severe disease, respectively, and a 43% efficacy against B.1.1.7 transmission. The most data points were available for estimates of efficacy against symptomatic disease after two doses (n = 13) ranging from 62.1% (95% CI: 41.0, 75.7) to 90.0% (67.4, 97.0) against the wild-type variant 6 . The latter estimate was from a low-dose/standard-dose regime which was excluded from our final evidence synthesis. We also excluded the most recent UK-based clinical trial from our synthesis which reported a vaccine efficacy against disease caused by non-B.1.1.7 variants of 84.1% (95% CI: 71 -91%). This was because inspection of the results indicated that over half of the samples in that post-hoc stratification were not sequenced and that efficacy was much lower in unsequenced samples 51 . In comparison, only one estimate of efficacy against severe disease after two doses was available compared to six estimates after one dose. Estimates for protection against transmission are still scarce with only two estimates available from the same study 14 . There were insufficient data to inform estimates of protection against transmission for wild-type or B.1.351. Finally, only one estimate of efficacy against B.1.351, against symptomatic disease after two doses (10.0%, 95% CI: -77.0, 55.0), was identified 52 .

Sputnik V (viral vector)
Only one eligible study was identified for Sputnik V (Figure 4) which reported an efficacy against wild-type symptomatic disease after one and two vaccine doses of 73.6% (95% CI: 13.1, 91.9) and 91.6% (95% CI: 85.6, 95.2) respectively 9 . This informed 2 of the 24 possible synthesised estimates (Underlying data 50 ). An effectiveness study reported a 79.4% efficacy for Sputnik Light, the first component of Sputnik V which is now authorised as a single dose product in Russia 53 . No data on efficacy against VOCs were available. Other endpoint estimates were assumed to be the same as AstraZeneca ChAdOx1, which is also a viral vector vaccine. We thus estimated a 92%, 92%, and 99% efficacy after two doses for wild-type against any infection, symptomatic disease, and severe disease respectively. There were not enough data to inform estimates of protection against transmission for wild-type or B.1.351.

Johnson & Johnson (viral vector)
Two studies reported efficacy and effectiveness estimates for the single dose Johnson & Johnson vaccine which informed 5 of the 12 possible estimates 54,55 ( Figure 5 and  55 . No data were available for efficacy against transmission, therefore we assumed this to be the same as for AstraZeneca. There were not enough data to inform estimates of protection against transmission for wild-type or B.1.351.

Pfizer-BioNTech (mRNA)
We identified 27 eligible studies on vaccine efficacy or effectiveness for the Pfizer vaccine ( Figure 6) which directly informed 14 of the 24 synthesised estimates (Table 2). We estimated 84%, 95%, and 95%, efficacy after two doses against wild-type for any infection, symptomatic disease, and severe disease, and 90% efficacy against B.1.1.7 transmission.
The majority of data available described estimates of efficacy against symptomatic disease after two doses (n = 15) ranging from 91.3% (95% CI: 89.0, 75.7) 56    were scarce with only one press release and a Qatari effectiveness study reporting estimates for B.1.351 56,58 . In our synthesis we assumed that the efficacy against severe disease reported by Vasilieou et al. was 80%, estimated from the weighted average (by the number of observations) of efficacy from 21 days onwards 59 .

Moderna (mRNA)
Available efficacy and/or effectiveness data for Moderna were not as comprehensive compared to Pfizer with only a single study reporting efficacy estimates for symptomatic (94.1%, 95% CI: 89.3, 96.8) and severe disease (100%, 95% CI: NA) after two doses for wild-type 5 (Figure 7). In our synthesis, we therefore used the mean value from effectiveness studies that did not differentiate between the mRNA vaccines (Pfizer or Moderna) to inform our estimates for wild-type against any infection after one and two doses, symptomatic disease after one dose, and severe disease after one dose (Table 2). We thus estimated 86%, 94%, 99%, and 90% efficacy after two doses against wild-type for any infection, symptomatic disease, severe disease, and transmission respectively.

Novavax (protein subunit)
We identified two eligible studies on vaccine efficacy for the Novavax vaccine (Figure 8) which informed 8 of the 24 estimates (Table 2). We estimated a 96%, 96%, and 99% efficacy after two doses against wild-type for any infection, symptomatic disease, and severe disease respectively. There were not enough data to inform estimates of protection against transmission for any non-wild type variant.

CoronaVac (whole virus)
We identified four eligible studies for the CoronaVac vaccine ( Figure 9) which informed 4 of the 24 possible estimates (Table 2). We estimated 67%, 67%, and 92% efficacy after two doses against wild-type for any infection, symptomatic disease, and severe disease. There were not enough data to inform estimates of protection against transmission for wild-type or B.1.351/P.1. The most datapoints were available for efficacy against symptomatic (n=6) and severe (n=6) disease ranging from 50.7% (95% CI:      the remainder assumed to be the same as CoronaVac (Table 2). We estimated a 67%, 78%, and 79% efficacy after two doses against wild-type for any infection, symptomatic disease, and severe disease respectively. There were not enough data to inform estimates of protection against transmission for any variant. No studies were identified reporting results for the WIBP vaccine.

Discussion
This review and evidence synthesis provides an overview of the currently available vaccine efficacy and effectiveness estimates for COVID-19 across multiple vaccine trials and platforms capturing the differences and considerable uncertainty in the protective impact of vaccines. We found the largest number of studies for the AstraZeneca and Pfizer vaccines whilst estimates for Sputnik V and Sinopharm were the most limited. For some vaccines, estimates relied on manufacturer press releases rather than peer-reviewed papers which limited its interpretation. Furthermore, efficacy estimates against VOCs, particularly B.1.351 and P.1 are limited. This is partly due to their more recent emergence compared to B.1.1.7, but also the lack of systematic sequencing in many affected countries. Thus variant-specific estimates are difficult to assess and are often reported as a mix of variants 10 . Our evidence synthesis suggests that efficacy estimates for the same vaccine platform are relatively consistent for wild-type, but there is greater variability for VOCs.
Assessment of efficacy and effectiveness of any vaccine is complex, especially for SARS-CoV-2 where our knowledge of the pathogen and the immune response is still evolving. Adding to this complexity is the emergence of new VOCs, some of which carry mutations allowing a degree of immune escape 24,63 . With COVID-19 vaccines being rolled out globally on an unprecedented scale, heterogeneities such as demographics and comorbidities between different populations will also impact vaccine effectiveness 57 . With some countries prioritising delivery of the first dose to as many individuals as possible over delivery of the second 18,20 , understanding how protection may differ after each dose of vaccine (for a multi-dose product) is also crucial. Furthermore, while most countries have focused on minimising COVID-19 mortality, or reducing pressures on the health system 64,65 by vaccinating the elderly, at least one country has prioritised younger adults who contribute most to transmission 66  A population-wide study from Israel found decreased viral loads amongst infected vaccinated individuals suggestive of a reduction in infectiousness 75 . Robust estimates of vaccine efficacy against infection and transmission will be vital in understanding the potential success of a reduced transmission strategy.
Most clinical trials were planned and undertaken before the emergence of VOCs and studies that specifically address efficacy against variants are sparse. Interpreting efficacy results for VOCs typically requires polymerase chain reaction (PCR)-sequencing of cases. If vaccination reduces viral loads in breakthrough cases, as suggested by clinical trials and effectiveness studies for AstraZeneca 51 78 . Especially for countries opting to delay delivery of the second dose in order to deliver as many first doses as possible, emerging data and continued assessment of vaccine effectiveness against VOCs are crucial in determining the optimal vaccination strategy.
The duration of follow-up in clinical trials and effectiveness studies is not yet sufficient to robustly estimate the duration of vaccine-induced immunity. How waning of immunity may affect the design of booster campaigns being considered to mitigate the potential impact of VOCs 79 will be a critical factor shaping the trajectory of the pandemic in the coming one or two years. Longitudinal cohort studies are needed to better assess the duration of both natural and vaccine-induced immunity.
Until results of forthcoming studies on efficacy against VOCs, transmission, and infection are available 80,81 , simulation studies must acknowledge and capture current uncertainty in order to robustly explore and inform vaccination policies and policy around the lifting of NPIs. Finally, it is important to note that this overview, at the time of writing, represents our best interpretation of the available efficacy and effectiveness data. As more results from clinical trials and effectiveness studies become available, available evidence should be re-evaluated.