How they’ll know if the COVID-19 vaccines work is a matter of math, biology — and a little luck

UPDATE: Pfizer Inc. said Monday morning that its vaccine has an efficacy of more than 90%, based on preliminary data. Though that is well above the company’s 60% initial projection, the company says the study will continue. The early results are based on 94 confirmed cases of COVID-19, while the initial plan was to wait for 164 confirmed cases.

Christine Staines works part-time in a gym, surrounded by huffing and puffing customers. So they take COVID-19 precautions seriously at Retro Fitness in Egg Harbor Township — keeping the door open for better ventilation, wearing masks at all times, and regularly cleaning the exercise equipment and other surfaces.

Staines herself may have an additional layer of protection: In late summer, she got two experimental injections that might prevent her from becoming infected.

Then again, they might not.

The Galloway, N.J. resident is among more than 42,000 people so far who have signed up for a trial of a vaccine developed by Pfizer Inc., in partnership with the German firm BioNTech SE. Half were chosen at random to get two doses of the vaccine, and half got two injections of saline solution — a placebo — yet no one was told which. Researchers will follow them for months to see how many in each group get sick.

This is the crux of any high-quality medical study: comparing what happens to people who get the drug with the outcomes in those who do not. That way, scientists can determine if any change is truly due to the drug and not some other factor, such as an underlying medical condition or just plain luck.

But vaccines are different in one key respect. Most participants in the various trials — provided they don’t change the social distancing and other habits that have kept them healthy so far — are unlikely to be exposed to the threat the drugs are designed to prevent. That means trial administrators must give the vaccines to a lot of people. And while they cannot ethically tell volunteers to let down their guard against a virus that has killed more than 230,000 Americans, researchers nevertheless count on at least a few hundred participants getting sick.

Challenge trials, in which people are deliberately exposed to the virus, are under discussion, though not in the U.S.

Staines, 47, said she has not changed her behavior since getting the shots. She eats at restaurants, but only outside. She went back to work Sept. 1 when New Jersey allowed gyms to reopen, but said the various precautions have, as far as anyone can tell, done the trick.

“If I’m going to get it, I’m going to get it,” she said.

She decided to participate in the trial after hearing from friends who work at the private medical practice where she signed up: South Jersey Infectious Disease in Somers Point. Plus an acquaintance of hers had been hospitalized with severe COVID-19 and was still in rehab months later. So she felt moved to do her part.

“If I could help to prevent somebody from being in that position, that’s the whole goal,” she said.

The quality that researchers are measuring for each vaccine is called efficacy: how much it reduces the risk of illness. (Efficacy describes the results in the controlled setting of a trial, when they can carefully pick how many people are in each group. A vaccine’s effectiveness, on the other hand, is an estimate of how well it works after approval, when anyone can get it.)

No vaccine prevents 100% of infections, though the measles vaccine comes close. Seasonal flu vaccines are often less than 50% effective — though they can reduce severity of illness in vaccinated people who do get the flu, which is a big deal if it means not going to the hospital. The COVID-19 vaccines must demonstrate efficacy of at least 50% to win approval, the U.S. Food and Drug Administration has said.

Based on pre-trial studies, Pfizer projects that the efficacy of its vaccine could be 60%. Working with that assumption, statisticians have calculated that out of an eventual total of 44,000 volunteers, at least 164 people must become infected for them to tell, with a reasonable degree of certainty, whether the vaccine works. If the efficacy truly is 60%, that means out of the 164 people, 117 would have gotten the placebo and 47 got the real thing. (47 is 60% lower than 117.)

Moderna, another vaccine maker whose candidate has reached the phase 3 stage of testing, projects it will need 151 of its 30,000 volunteers to get sick. If the true efficacy is 60%, that means 108 placebo recipients would get sick, while just 43 in the vaccine group would do so.

But these are population samples, like those taken in a political poll. Neither trial is likely to hit those figures exactly, said former FDA biostatistician Susan S. Ellenberg, a professor at the University of Pennsylvania’s Perelman School of Medicine.

“You have to have a number of cases that’s big enough so you can have confidence in your estimate,” she said.

Yet if the vaccines work better than anticipated, both companies say that might become apparent before reaching those numbers of infections — in which case, they would seek the FDA’s approval to distribute them on an emergency basis.

Bad idea, said Marcus Schabacker, chief executive officer of ECRI, a Plymouth Meeting-based nonprofit research institute that evaluates the quality and cost of medical treatments.

The first few thousand people who enroll in each trial might not be representative of the broader population in some way, such as age, gender, ethnicity, or overall health, he said. Moderna’s trial, for example, was slow at first to enroll people of color, though a recruitment effort boosted their representation to 37%.

If people in any overrepresented subgroups are more or less likely to get sick, it could skew the data, making the vaccine seem either more or less potent than it truly is, Schabacker said. Better to wait for the full enrollment.

“These trials have been designed well,” he said. “Let’s stick to the plan.”

What’s more, if the trials were cut short after two or three months, scientists could not tell if a vaccine’s protection were to wear off in months four or five, said Schabacker, an anesthesiologist and critical-care specialist. Another wrinkle: the vaccine may confer partial protection for some people, meaning they could still be infected but with milder symptoms. That would be good, unless it meant they were unaware of being contagious and therefore passed it on to others.

Along with efficacy, trial overseers must watch for any "adverse events” in trial participants and judge whether they might have been caused by a vaccine. That is yet another reason not to cut any trials short, as any failure to identify a grave side effect — eroding public trust in vaccine safety — would be disastrous, Schabacker said.

And this surveillance must continue even after approval, said Penn’s Ellenberg. If one of the vaccines provokes an unforeseen reaction in, say, 1 out of 10,000 recipients, on average, then it very possibly might not show up in an initial trial group of 15,000 or 20,000 people, she said.

It all takes time. One way that some trial sites are trying to speed up the process is by giving the vaccines to health-care workers and others who may be more likely to be exposed.

Among them is Robert L. Kruse, a participant in the Pfizer trial and a pathology resident at Johns Hopkins Medicine. But mask-wearing and other precautions at his hospital are so rigorously enforced that his risk is probably low, he said.

“You can’t walk in without someone making sure you have everything on,” he said.

Another way to get faster results is something called a “challenge” study, in which vaccine recipients are deliberately infected with the virus. Researchers in Great Britain have proposed conducting such a study using young, healthy volunteers, for whom the risk of death is extremely low. But medical ethicists have been critical, warning that even when it does not kill, the virus can have severe, lingering consequences for some.

» HELP US REPORT: Are you a health care worker, medical provider, government worker, patient, frontline worker or other expert? We want to hear from you.

For now, the U.S. trials depend on the chance that a few hundred volunteers like Kruse and Staines will get sick. Given that they are being conducted amid a pandemic, the trials are bound to be unpredictable, Kruse said.

“This is the gold standard," he said, "but they’re relying on these factors outside of their control.”

Yet with cases on the rise in much of the country now, as the weather grows colder and people gather indoors, perhaps answers will come sooner than he thinks.