With all the questions in the news we asked experts Matt Keeling and Sam Moore, epidemiological modellers at Warwick University and members of the JUNIPER modelling consortium, how the COVID-19 vaccines were tested. Here is what we learned.

All medical drugs, including vaccines, go through various phases of testing (find out more here) to make sure they are safe and worthy enough of large-scale randomised controlled trials (RCTs) in humans. RCTs involve a large number of people, picked at random to make sure they are representative of the population you want to use the drug on — the randomisation makes sure we don't end up with people who are predominantly of a certain age, or gender, or predominantly display other characteristics that might confuse the results of the study. However, often groups who are at a particularly high risk may be excluded from early trials, as we may want to first test the drug or vaccine on healthy volunteers. People from such groups will not be given the vaccine until it is proven to be safe for them.

The trial participants are then randomly allocated to two groups: one group gets the drug (vaccine) you want to test and the other group gets a neutral substance, or a placebo. The latter is done to deal with the famous placebo effect which has people feeling an effect of a treatment only because they know they have been given it, even if the treatment doesn't actually work. Scientists want to make sure that the drug they are testing is more effective than the placebo effect — and has no worse side effects.

Obviously, participants of the trial aren't told whether they are in the group receiving the drug to be tested (the study group) or in the group receiving another drug or the placebo (the control group). In a double-blinded trial, the people who administer the drug also don't know whether the person they are giving the drug to is in the study group or the control group. And in a triple-blinded trial, even the scientists doing the statistical analysis of the results don't know who is in which group. Blinding is done to make sure that no form of bias creeps in through a person consciously or unconsciously skewing the outcome into some direction.

In an RCT the health of the people in the study group and the control group is being monitored.

Once the people have been given their drugs or the placebo, you see how many register an effect as a result. If many more people register an effect in the study group than in the control group, then that's good evidence that your drug works. For vaccine trials, we want to observe that fewer people who were given the vaccine get ill (with the disease) compared to those in the control group.

However, there's still the possibility that such a result was a fluke. The drug or vaccine may have been more effective in the study group than in the control group by pure chance, or due some other underlying cause that has nothing to do with the drug or vaccine. Probability theory enables you to work out the probability of that happening (see this article to find out more). If that probability is small — typically the threshold is less than 5% — then you decide that the drug actually does work.

It's very important in any randomised controlled trial (RCT) that the sample size, the number of people taking part in the trial, is large enough: if you only test a drug on four people, then clearly the results won't give you reliable information. Here's where testing a vaccine, rather than another type of drug, becomes a bit tricky. Ideally (from a testing point of view) you would expose all your trial participants to the virus and then see how many of them develop COVID-19 and how many don't. But that clearly wouldn't be ethical. Instead, you have to wait until people in your control group catch COVID-19 of their own accord — and that number of infected people needs to be big enough if your trial is to have sufficient statistical power. But how do you control this number?

The answer is that you need to start with a large enough initial sample of people to be sure the number of COVID-19 infections ends up large enough. Luckily, working out the size of this initial sample isn't guesswork, but comes from a calculation involving the statistics of COVID-19. One important number in this context is the incidence of the disease in the population: the more disease there is around, the more people in your control group will catch it. This is why parts of the trials of the Pfizer/BioNTech and the Oxford/AstraZeneca vaccines were conducted in places with high incidence, such as Brazil.

Another important number is the effectiveness (or efficacy) of the vaccine: the less effective you think it will be, the more people you need to catch COVID-19 in the control group to be able to discern the effect of the vaccine in the study group. A statistical calculation involving these numbers then tells you how big your initial sample needs to be.

As we said above, all the necessary steps in the testing process were followed for both the Pfizer/BioNTech and the Oxford/AstraZeneca vaccine, and have been independently verified, so we can be sure that the conclusions are correct.

Return to our COVID-19 vaccines FAQ

About this article

Matt Keeling

Matt Keeling is a professor at the University of Warwick, and holds a joint position in Mathematics and Life Sciences. He is the current director of the Zeeman Institute for Systems Biology and Infectious Disease Epidemiology Research (SBIDER). He has been part of the SPI-M modelling group since 2009.

Sam Moore is a postdoctoral research associate who has been working on vaccination modelling for Covid-19 after joining SBIDER within the University of Warwick at the start of the pandemic earlier this year.

Sam Moore

Both are members of JUNIPER, the Joint UNIversity Pandemic and Epidemic Response modelling consortium. It comprises academics from seven UK universities who are using a range of mathematical and statistical techniques to address pressing question about the control of COVID. The universities are Cambridge, Warwick, Bristol, Exeter, Oxford, Manchester, and Lancaster. You can see more content produced with JUNIPER here.

Marianne Freiberger and Rachel Thomas are Editors of Plus.