Evaluating COVID vaccines: Clinical trials vs. observational studies

As more and more Canadians get their COVID-19 shots, some may be wondering how long that protection will last. Scientists have proven that COVID-19 vaccines are highly effective over the short term in clinical trials. But, as with all newly approved vaccines, we don’t know how effective these vaccines will be over time.

As new variants emerge and public health guidelines continue to evolve, researchers must use different ways to evaluate COVID-19 vaccines in the field.

Clinical trials are necessary for vaccine approval. They assess vaccine safety, efficacy and immunogenicity (or how the body’s immune system responds to a vaccine). But a different set of epidemiological methods – known as observational studies – are needed to know how well a vaccine works once it’s rolled out in the population.

What are clinical trials?

Clinical trials are types of studies that scientists use to investigate new drugs or interventions, like vaccines. They are often referred to as RCTs or randomized controlled trials. Randomized because individuals are randomly assigned to receive an intervention – either a vaccine or placebo in most of the COVID-19 vaccine trials – and controlled because individuals who do not receive the intervention act as a reference group.

Clinical trials for several COVID-19 vaccine candidates are ongoing. These trials are investigating new vaccines that Health Canada has not yet approved. Some manufacturers have also expanded their trials of already-approved vaccines to include different populations (e.g., children younger than age 12).

Why are observational studies needed if Health Canada has already authorized COVID-19 vaccines?

Information about how vaccines work in clinical trials may not apply to real-world situations. People who participate in clinical trials are often young, healthy volunteers who can respond differently to vaccines than the broader group of people who get vaccinated.

“The clinical trials were based on relatively small, selected populations,” explains Jeff Kwong, a professor and interim director of the Centre for Vaccine Preventable Diseases at the University of Toronto. “What happens when you get out into the real world? We don’t really know. There’s all sorts of factors, like how the vaccine is given in the real world, that may be slightly different.”

COVID-19 vaccines have now been authorized for use in many countries, including Canada, based on clinical trial results. Evidence from these trials shows that they are safe and highly effective, preventing up to 95 per cent of symptomatic COVID-19 illness.

However, because public health experts recommend COVID-19 vaccines for all eligible Canadians – and with close to 80 per cent now vaccinated with at least one dose – some experts believe that it is no longer ethical or feasible to ask participants in clinical trials to accept the risk of getting a placebo instead of an effective vaccine.

For this reason, observational studies are needed to find out how well COVID-19 vaccines work in the real world and fill in the remaining knowledge gaps from clinical trials. The World Health Organization recently published guidelines on how to do this.

“These observational studies are an opportunity for us to look to see if these vaccines work as well in all patient groups, in the entire population, and not just the clinical trial population,” says Kwong.

What can observational studies tell us that we don’t already know from clinical trials?

“Vaccine effectiveness studies are really important for complementing and building on the evidence that we’re getting from Phase 3 clinical trials,” says Nicole Basta, an associate professor at McGill University who studies vaccines. “They can really help us answer more in-depth questions, questions in different populations, different settings and different epidemiological contexts.”

For example:

• What is the duration of protection from vaccines and their long-term vaccine effectiveness?
• How well do these vaccines work in specific sub-groups who were excluded from the trials like pregnant women or people who have compromised immune systems?
• What is the vaccine effectiveness against different variants of concern and will booster doses be needed?

“Clinical trials answer a very particular question about how well the vaccines work for individuals,” says Basta. “But population-based studies and post (licensure) vaccine effectiveness studies are really asking a different question about a population.”

Understanding the impact of our vaccination programs is critical in situations of limited supply in which vaccine products may be used off-label from what Health Canada had originally approved.

As variants emerge and guidelines evolve, researchers must use different ways to evaluate COVID-19 vaccines.

For example, researchers in Canada and elsewhere have used observational studies to confirm the “first dose fast” strategy, whereby a person’s second vaccine dose is delayed so that more people can get a first dose. “This is something that we couldn’t detect in the clinical trials because they had their dosing interval so short,” says Kwong, who is the lead author of one of these studies. “I think this is probably the most policy-relevant aspect (of observational studies).”

Other examples include evaluating mixed dose strategies, in which an individual receives different vaccine products for first and second doses, or comparing vaccination impacts between jurisdictions that adopted different strategies.

Why might vaccine effectiveness estimates in the field differ from clinical trials?

Clinical trials measure vaccine efficacy against well-defined clinical outcomes under the strict conditions of a trial. Observational studies measure vaccine effectiveness in real-world field settings. Both types of studies compare the rate of disease in the group of people who are vaccinated to those who are unvaccinated.

Estimates of vaccine efficacy from clinical trials can be higher or lower than what is seen in the field. That’s because clinical trials have strict criteria for who can be included in the study and only follow people for a short period of time.

Although these trials enrolled large numbers of volunteers, they had too few people who were hospitalized or died to precisely measure vaccine efficacy against these more severe outcomes. They also weren’t able to study how well the vaccines stopped SARS-CoV-2 transmission or prevented asymptomatic illness. Instead, they only looked at infections in people who developed symptoms.

Most of these trials were conducted during the summer and fall of 2020 before certain variants of concern first emerged. They also didn’t account for programmatic issues in the vaccine rollout like limited vaccine supplies or cold storage requirements, nor public health guidelines that are constantly changing due to evolving scientific evidence.

Scientists can use observational studies to answer questions about waning immunity (when vaccine protection declines over time), as well as the indirect effects of herd immunity.

“Phase 3 clinical trials are focused typically on one question, and that is evaluating what the efficacy is against symptomatic disease, and sometimes also against more severe outcomes like hospitalizations and death,” says Basta. “Vaccine evaluations that happen at the population level (…) give us the opportunity to expand the range of questions that we’re asking about COVID-19 vaccines.”

How do you prevent biases in observational studies?

In clinical trials, the intervention is under the control of the scientists who are leading the trial. In contrast, in observational studies individuals make informed choices about whether or not to get vaccinated.

Because people who get vaccinated may have a different risk of COVID-19 exposure or infection due to extraneous factors that also affect their access to health care and testing, vaccinated and unvaccinated groups are not directly comparable in observational studies. This imbalance – or what epidemiologists call confounding – can lead to biased study results.

“(Observational) studies are really challenging to do and to do well because you need to make sure that the comparison groups are as similar as possible at baseline,” says Basta.

Because clinical trials assign individuals to different intervention groups randomly by design, they can more easily make sure that any potential confounders are evenly distributed between the groups.

“I would say that an observational study done well is comparable to a clinical trial. An observational study done poorly is worthless,” says Basta. That’s why it’s important to drill down into the methods of observational studies and find any potential sources of bias (see the chart below). She also encourages researchers to acknowledge uncertainty around their findings.

“I think this is where having multiple studies in multiple epidemiological contexts is really important,” says Basta. “We can have each study build upon each other and start getting a bit of a consensus about how well the vaccines work.”