One weird thing that’s been holding drug trials back

Here’s something strange about how we test new drugs: Every clinical trial has to pretend that nothing like it has ever come before.

Even if clinicians have tested similar drugs for years, or if decades of research point in a certain direction, each trial must prove — independently — that the drug works based solely on what happens inside that specific study. Prior knowledge doesn’t count.

For more than 60 years, this blank slate approach has been the Food and Drug Administration’s gold standard — and for good reason. If you let prior research formally count toward proving a drug works, drug companies might easily cherry-pick the studies that flatter their results.

Naturally, such rules have led to academic circle jerks over whether past research should factor into the final verdict on a drug. But for patients, the cost of starting from scratch every time can be high.

For people with rare diseases, where only a few hundred individuals worldwide might have a condition, running a traditional trial can be nearly impossible, because there simply aren’t enough patients to enroll. For children, it has meant re-proving what we already learned in adults. And for everyone, it has meant slower, more expensive trials that throw away useful information.

Now, the FDA is telling drug companies and researchers they don’t have to start from scratch anymore.

Last week, the agency released new guidance encouraging companies to use a statistical approach, that would usually be used on a case-by-case basis, called Bayesian methods. (We’ll get more into that later.)

What that means is that, for the first time, companies can formally incorporate what they already know — from earlier studies, from related drugs, from real-world evidence — to help answer the central question of whether a drug works. The FDA’s guidance is still a draft, and details may shift over the coming months, but the policy signal is clear.

“It sounds so intuitive to just use the data that you have before to inform the next thing that you do,” said Merit Cudkowicz, a neurologist at Massachusetts General Hospital who runs a major ALS clinical trial, “instead of just having this sort of amnesia.”

Two ways of looking at the world

For a drug to get FDA approval, it has to prove it works in three phases of clinical trials. But “proving it works” can mean different things, depending on how you handle uncertainty.

The traditional approach — called frequentist statistics — asks a narrow question: If this drug doesn’t actually work, how likely is it that we’d see results this strong just by chance? If that probability is very low (typically below 5 percent), the drug passes the test. The appeal is objectivity; the trial data speaks for itself, and what you believed going in doesn’t formally enter the math.

Bayesian statistics, the new rule of the land, flips the question. It asks: Based on everything we already know, how likely is it that this drug works? Then, it updates that estimate as new trial data comes in. The result isn’t a binary pass/fail, but a probability — say, a 94 percent chance the drug is effective. That doesn’t mean anything goes, and the FDA still has to draw a line in the sand that’s pre-agreed before the trial runs.

The practical upshot is that Bayesian methods let you formally “borrow” information from other places. If you’ve already tested a drug in adults, you can use that data when evaluating it in children. If you’re running a trial with multiple drugs, data from one arm of the study can inform another. This flexibility matters most in situations where patients are hard to come by.

“The availability of prior information is why we see such use in pediatric,” said James Travis, a statistician in the FDA’s drug review division. “We pretty much always have adult information, so it’s very easy to do things like that in the pediatric space.”

But being able to bring in outside information raises one obvious concern: What is stopping researchers from cherry-picking the studies that make their drug look good?

Traditional trials have a hard threshold — the “p-value,” a measure of whether results are likely due to chance — that seems to remove human judgment out of the equation. You either hit statistical significance, or you don’t. Bayesian methods, by contrast, require researchers to choose “priors,” or assumptions about what they expect to find based on existing evidence.

But this critique assumes that traditional trials are capital-O objective, and that’s not necessarily the case; they just hide their assumptions better.

Every clinical trial involves choices: which patients to enroll, what outcomes to measure, what comparisons to make. A p-value can make it seem like the math is deciding, when, in fact, subjective judgments are baked in throughout.

Bayesian methods, proponents argue, force those assumptions into the open. You have to state your priors upfront, and justify them. And then everyone — including FDA reviewers — can see exactly what you assumed and evaluate whether it was reasonable.

Why patients care about statistics

All of this might sound like an academic statistical debate. But for people with serious diseases and their loved ones, the stakes are stark.

Consider amyotrophic lateral sclerosis (ALS), a neurodegenerative disease that kills most patients within two to five years of diagnosis. Around 5,000 Americans are diagnosed each year, according to the Centers for Disease Control and Prevention’s National ALS Registry.

But despite decades of research, drug trials kept failing. Testing one drug at a time, starting essentially from scratch each time, was painfully slow for a disease that doesn’t have much wait time.

In 2019, the FDA greenlit an unusually Bayesian trial to hunt for new ALS drugs. In the HEALEY ALS Platform Trial, researchers at Massachusetts General Hospital were able to test multiple ALS drugs at once, fast enough to matter for patients who didn’t have time to wait. Data from patients in one part of the trial — including those receiving placebos — can be used to inform drugs in other parts of the large-scale trial. This means the trial can drop drugs that aren’t working and add promising ones without starting over each time.

In the four years the trial has been running, seven drugs have been tested so far. A traditional approach might have managed just two. The new FDA statistical guidance, Cudkowicz said, should clear the path for other trials to follow this sort of model.

“The patients enrolled so fast because the patients with ALS felt that this was a patient-centered trial,” said Merit Cudkowicz, the neurologist who leads the study. Two of those drugs showed enough promise that they’re now advancing to final-stage trials.

“The Bayesian approach is just trying to take all of that data that participants give – and they give a lot of themselves – and use it in the most effective way,” said Melanie Quintana, a statistician at Berry Consultants, who helped design the HEALEY trials.

More flexibility also means more room for things to go wrong.

A 2018 review, co-authored by Aaron Kesselheim, a Harvard professor who studies FDA policy, examined more than 100 adaptive trials, a related approach that also allows mid-trial adjustments and often uses Bayesian methods. They found that only a third of trials used independent committees to monitor the data, and just 6 percent kept statisticians blinded when analyzing mid-trial. Without these safeguards, there’s more room for bias to creep in or for early results to mislead.

FDA officials say the safeguards for Bayesian trials will remain. Every proposal will be reviewed by agency statisticians, and companies must lock in their methods before the trial starts.

“It’s not like you get to pick the prior after you’ve seen the data,” John Scott, who oversees biostatistics at the FDA. “There’s really strict rules about that.”

But whether individual companies actually start using these methods is another question. The guidance is not yet set in stone. The proposal is open for public comment until March 13, with a final version expected in about 18 months. And with FDA facing leadership turnover and political uncertainty, companies may be even more cautious about trying something new.

“Drug companies hate uncertainty,” said Adam Kroetsch, a former FDA official who has written about the agency’s evolution. “They might decide it’s not worth the risk and just go with the traditional approach where they know there’s FDA precedent.”

But the FDA isn’t alone in this shift – the European Medicines Agency has also been exploring expanded use of Bayesian methods in drug development.

For patients with rare diseases, or for children waiting on treatments that already work in adults, the stakes of this statistical change are potentially life or death. The HEALEY trial has already shown what’s possible, and the FDA has opened the door. Now, more companies have to walk through it.