Is the FDA Too Conservative or Too Aggressive?

by on August 26, 2015 at 7:24 am in Economics, Medicine, Science | Permalink

I have long argued that the FDA has an incentive to delay the introduction of new drugs because approving a bad drug (Type I error) has more severe consequences for the FDA than does failing to approve a good drug (Type II error). In the former case at least some victims are identifiable and the New York Times writes stories about them and how they died because the FDA failed. In the latter case, when the FDA fails to approve a good drug, people die but the bodies are buried in an invisible graveyard.

In an excellent new paper (SSRN also here) Vahid Montazerhodjat and Andrew Lo use a Bayesian analysis to model the optimal tradeoff in clinical trials between sample size, Type I and Type II error. Failing to approve a good drug is more costly, for example, the more severe the disease. Thus, for a very serious disease, we might be willing to accept a greater Type I error in return for a lower Type II error. The number of people with the disease also matters. Holding severity constant, for example, the more people with the disease the more you want to increase sample size to reduce Type I error. All of these variables interact.

In an innovation the authors use the U.S. Burden of Disease Study to find the number of deaths and the disability severity caused by each major disease. Using this data they estimate the costs of failing to approve a good drug. Similarly, using data on the costs of adverse medical treatment they estimate the cost of approving a bad drug.

Putting all this together the authors find that the FDA is often dramatically too conservative:

…we show that the current standards of drug-approval are weighted more on avoiding a Type I error (approving ineffective therapies) rather than a Type II error (rejecting effective therapies). For example, the standard Type I error of 2.5% is too conservative for clinical trials of therapies for pancreatic cancer—a disease with a 5-year survival rate of 1% for stage IV patients (American Cancer Society estimate, last updated 3 February 2013). The BDA-optimal size for these clinical trials is 27.9%, reflecting the fact that, for these desperate patients, the cost of trying an ineffective drug is considerably less than the cost of not trying an effective one.

(The authors also find that the FDA is occasionally a little too aggressive but these errors are much smaller, for example, the authors find that for prostate cancer therapies the optimal significance level is 1.2% compared to a standard rule of 2.5%.)

The result is important especially because in a number of respects, Montazerhodjat and Lo underestimate the costs of FDA conservatism. Most importantly, the authors are optimizing at the clinical trial stage assuming that the supply of drugs available to be tested is fixed. Larger trials, however, are more expensive and the greater the expense of FDA trials the fewer new drugs will be developed. Thus, a conservative FDA reduces the flow of new drugs to be tested. In a sense, failing to approve a good drug has two costs, the opportunity cost of lives that could have been saved and the cost of reducing the incentive to invest in R&D. In contrast, approving a bad drug while still an error at least has the advantage of helping to incentivize R&D (similarly, a subsidy to R&D incentivizes R&D in a sense mostly by covering the costs of failed ventures).

The Montazerhodjat and Lo framework is also static, there is one test and then the story ends. In reality, drug approval has an interesting asymmetric dynamic. When a drug is approved for sale, testing doesn’t stop but moves into another stage, a combination of observational testing and sometimes more RCTs–this, after all, is how adverse events are discovered. Thus, Type I errors are corrected. On the other hand, for a drug that isn’t approved the story does end. With rare exceptions, Type II errors are never corrected. The Montazerhodjat and Lo framework could be interpreted as the reduced form of this dynamic process but it’s better to think about the dynamism explicitly because it suggests that approval can come in a range–for example, approval with a black label warning, approval with evidence grading and so forth. As these procedures tend to reduce the costs of Type I error they tend to increase the costs of FDA conservatism.

Montazerhodjat and Lo also don’t examine the implications of heterogeneity of preferences or of disease morbidity and mortality. Some people, for example, are severely disabled by diseases that on average aren’t very severe–the optimal tradeoff for these patients will be different than for the average patient. One size doesn’t fit all. In the standard framework it’s tough luck for these patients. But if the non-FDA reviewing apparatus (patients/physicians/hospitals/HMOs/USP/Consumer Reports and so forth) works relatively well, and this is debatable but my work on off-label prescribing suggests that it does, this weighs heavily in favor of relatively large samples but low thresholds for approval. What the FDA is really providing is information and we don’t need product bans to convey information. Thus, heterogeneity plus a reasonable effective post-testing choice process, mediates in favor of a Consumer Reports model for the FDA.

The bottom line, however, is that even without taking into account these further points, Montazerhodjat and Lo find that the FDA is far too conservative especially for severe diseases. FDA regulations may appear to be creating safe and effective drugs but they are also creating a deadly caution.

Hat tip: David Balan.

1 prior_approval August 26, 2015 at 7:38 am

‘In the latter case, when the FDA fails to approve a good drug people die but the bodies are buried in an invisible graveyard.’

We will all die, regardless of what drugs are approved or not.

‘The bottom line, however, is that even without taking into account these further points, Montazerhodjat and Lo find that the FDA is far too conservative especially for severe diseases.’

Like this? – ‘Breakthrough Therapies (2013)

The Food and Drug Administration Safety and Innovation Act (FDASIA) includes a provision that allows sponsors to request that their drug be designated as a Breakthrough Therapy. A breakthrough therapy is a drug:

intended alone or in combination with one or more other drugs to treat a serious or life threatening disease or condition and
preliminary clinical evidence indicates that the drug may demonstrate substantial improvement over existing therapies on one or more clinically significant endpoints, such as substantial treatment effects observed early in clinical development.

If a drug is designated as breakthrough therapy, FDA will expedite the development and review of such drug. All requests for breakthrough therapy designation will be reviewed within 60 days of receipt, and FDA will either grant or deny the request.’ http://www.fda.gov/ForPatients/Illness/HIVAIDS/Treatment/ucm134331.htm

2 Jeff R. August 26, 2015 at 11:33 am

We will all die, regardless of what drugs are approved or not.

Right. That’s why I oppose universal healthcare. People will die anyway, so what’s the point?

3 Payam August 26, 2015 at 2:10 pm

+1

4 Jamie_NYC August 26, 2015 at 1:52 pm

“We will all die…”

Speak for yourself – I’m signing up with ALCOR.

5 rayward August 26, 2015 at 8:13 am

Of course, every terminally ill person believes, must believe, in a miracle cure, if only the FDA would allow it to market. Preying on the desperate, be they terminally ill or otherwise, is what charlatans do. Tabarrok’s “bottom line” is that the terminally ill will die anyway, so let them have the drugs being peddled by the charlatans. Because markets know better than the FDA. The Straussian Paradox: markets know better than people and certain people (themselves) know better than everybody else.

6 wiki August 26, 2015 at 8:25 am

Your kneejerk dislike of markets is blinding you to the fact that the market isn’t invoked in this study. The fact is that from a mathematical perspective, the cutoff for “significant” findings is arbitrary and Type II error is rarely considered during clinical trials. Thus, a purely analytic analysis of the FDA’s methods by an ideal central planner — let’s call him Rayward Lenin — would show that using the same cutoff for milder and graver diseases is folly. With or without free markets, such a standard keeps out drugs where the costs of non-approval are (probabilistically) much worse than approving a few drugs mistakenly.

7 dearieme August 26, 2015 at 8:41 am

I dare say that Lenin would disapprove of the heterogeneity point too: people aren’t individuals, are they? They’re just essentially identical
interchangeable items.

8 Joe Torben August 26, 2015 at 10:53 am

Wow. Just wow. Beta blockers were delayed more than ten years in the US. Many, many thousands of people died much earlier than if they would have had access to that medicine. It is certainly true that everyone would have died at some point anyway, but suggesting that allowing beta blockers is “preying on the desperate” may be the most cynical, evil and ignorant thing I will read this week. And I browse the internet daily, so that is saying a lot.

You are a terrible person who hate other people. That is fine, I guess, but it would be much preferred by everyone if you could avoid spewing your hate here. Thanks!

9 RB August 26, 2015 at 2:59 pm

Talk about an ad hominem attack!

10 Jeff R. August 26, 2015 at 11:31 am

“…every terminally ill person believes, must believe, in a miracle cure..”

That simply isn’t true. People faced with terminal illnesses do not suddenly lose their ability to think rationally.

11 Art Deco August 26, 2015 at 12:10 pm

People faced with terminal illnesses are not necessarily all that attached to this life, either. Or lacking in dignity.

12 Art Deco August 26, 2015 at 12:08 pm

Of course, every terminally ill person believes, must believe, in a miracle cure, if only the FDA would allow it to market

Well, we know one regular participant here would not know a terminally ill person from a cord of wood.

13 rayward August 26, 2015 at 12:27 pm

George Washington, a smart man, allowed his physicians to bleed him to death. Thomas Jefferson, a smarter man, once quipped: whenever two or more physicians gather, buzzards can’t be far. The belief in witch doctors, despite all the evidence, amazes me. Far more people die from over-medication than from under-medication. Even the medical statistics lie. We are told again and again that the survival rates for cancer patients has increased substantially. What the statistics don’t reveal is that “survival” is measured from the date of diagnosis, not in longevity, and in a nation obsessed with early diagnosis, it’s certainly true that “survival” has increased. One of the many ironies of this web site is the belief that “experts” should not intervene in matters of economics (or in getting drugs to market), but should intervene in matters of health.

14 Jeff R. August 26, 2015 at 12:56 pm

You’re an anti-vaxxer, aren’t you?

15 John Venable August 26, 2015 at 1:53 pm

I think a decent number of us might do away with the experts (the FDA) altogether, but that’s not what the article is about is it.

16 RB August 26, 2015 at 2:58 pm

+1
I agree with all of rayward’s comment, and I am very strongly pro vaccine!
There’s a large and growing literature on biases in the drug pipeline and FDA approval process, most of which are heavily in favor of approving drugs that do nothing (and yet, still have side effects). To mention one of many problems, the population used to test drugs is younger, healthier, more homogeneous, and more compliant than the population that ends up actually taking the drug. A second problem is that the testing process screens out people who have MAJOR side effects – they stop taking the drug, and are dropped from the sample (and from the statistical analysis at the end). So we only see the people with moderate side effects.

The paper is interesting, but it is working from an idealized model of the drug research process.
Check out Marcie Angel and more recent authors. The Truth About the Drug Companies: How They Deceive Us and What to Do About It. 2004. ISBN 9780375508462.

17 RB August 26, 2015 at 3:01 pm

Correction: I agree with rayward’s comment that starts “George Washington….” His other remarks are not so clear cut.

18 Marge I. Nal August 27, 2015 at 4:39 pm

“One of the many ironies of this web site is the belief that “experts” should not intervene in matters of economics (or in getting drugs to market), but should intervene in matters of health.”

I can’t speak for the entire “web site,” but that is exactly what this article is not “belie[ving].” I read the whole thing, and it appears to me that the author is advocating more personal decision-making (the “Consumer Reports” model, linked). Even if you consider the “off-label prescribing” to be done by an “expert” (the doctor), it is still one fewer level of “expert” than having it go through the FDA first.

This just sounds like basic trolling (in this case by mischaracterizing the argument and bringing in irrelevant arguments).

19 CCL2 August 26, 2015 at 12:16 pm

“…certain people (themselves) know better than everybody else.”

Only individuals exist, think, and act. The FDA is merely a small group of politically appointed individuals merging their personal judgemments/opinions… and forcibly imposing them upon all persons in the U.S. How is it rationally possible for the FDA to “know best” ?

Individuals can and should make their very own drug & health choices — for themselves. They can gather all the government, market, scientific, mystical advice they want, or voluntarily accept the health decisions/advice of another individual. The individual ultimately bears the consequences of his own health decisions, if at liberty to do so.

The FDA Commissars bear no consequences for dictating drug & health decisions to hundreds of millions of people… and they are no wiser than other Americans nor other sectors of society. That is the fundamental problem — not the FDA’s current statistical biases and whims.

20 Art Deco August 26, 2015 at 12:24 pm

The FDA is merely a small group of politically appointed individuals merging their personal judgemments/opinions… and forcibly imposing them upon all persons in the U.S. How is it rationally possible for the FDA to “know best” ?

The vast majority are civil service and professional appointees. The FDA commissioner is commonly a physician.

21 chuck martel August 27, 2015 at 6:12 am

” The FDA commissioner is commonly a physician.”

So what?

22 Jan August 26, 2015 at 4:12 pm

Stop being so dramatic. Getting the risk/benefit balance right, especially for subgroups of patients who might be willing to tolerate more or less risk than typical patients, is a big deal for FDA.

FDA knows there is potential for overly conservative decision making on their part and they have a whole initiative on just this topic–getting the patient perspective right and incorporating it into their drug evaluations.

http://www.fda.gov/ForIndustry/UserFees/PrescriptionDrugUserFee/ucm326192.htm

23 Ethan Bernard August 26, 2015 at 12:48 pm

“Of course, every terminally ill person believes, must believe, in a miracle cure…”

This reminds me of the canard “There are no atheists in foxholes.” Neither are true. Check out the videos of Christopher Hitchens in his last weeks for a counterexample.

24 Art Deco August 26, 2015 at 2:23 pm

It should not remind you of that. Atheists are atypical in most social circumstances other than elite circles and among people who manipulate words and imagery for a living. His notion that ‘every terminally ill person believes…’ is the remark of someone who has never set eyes on a terminally ill person or is basing his blather on an oddball sample of one.

25 Mark Bahner August 27, 2015 at 11:32 pm

No bureaucrat in Washington should ever be allowed to decide what can or cannot go into a patient’s body. That decision should always be the patient’s. (With advice from his or her doctors.)

26 jdm August 26, 2015 at 8:28 am

Without denying the FDA’s obvious incentives to be overly cautious, could you give a few recent examples of life-saving drugs that were unduly delayed by the FDA?

Likewise do you know of any relatively recent (say since 2000) examples of life-saving drugs where subsequent research made it clear that the FDA had made a Type-II error, rejecting a good drug? I realize that in many cases, we would not be able to tell if an FDA rejected drug is in fact effective because the rejection, as you note, will tend to curtail further research and investment in the drug, but there surely must be examples of rejected drugs that have had a second chance and were later vindicated.

I’m genuinely curious about this, not trying to score rhetorical points.

27 Marcos August 26, 2015 at 10:30 am

No, people won’t be able to tell you type II errors. That’s the entire point, type II errors are invisible.

Besides, the main effect of allowing a bigger probability of type II errors is a decrease on testing costs. The final status of the drugs change much less. In short, the article’s argument is that we should make testing cheaper and faster for drugs for terminal diseases, as there’s relatively little problem if the drug turn out to be ineffective.

28 jdm August 26, 2015 at 10:40 am

I can’t deny your concluding sentence, but if we cannot identify any type II errors, or identify any efficacious drugs that have been unduly delayed by the FDA, it’s a bit hard to determine how bad the problem that Alex is concerned with really is.

Besides, as Ed notes below, if there really are lots of good drugs that are being incorrectly rejected or delayed by the FDA (because they impose too stringent a limit on Type-II errors), wouldn’t we expect to see at least some of these drugs approved elsewhere, and thereby be able to point to a few obvious examples of recent FDA failures in this area? If so, what are the examples?

29 John August 26, 2015 at 10:54 am

You can’t identify future type II errors, but certainly you can identify past ones. Are there drugs that suffered excessive delays and then when approved suddenly started saving lives?

30 jdm August 26, 2015 at 11:12 am

Exactly.

31 Moreno Klaus August 26, 2015 at 10:42 am

If FDA would follow the argument article the pharmaceutical companies would flood the market with lower quality medicines for terminal diseases. And lets not forget the problem with using p-values for multiple testing (lets say a company would test multiple drugs until it passes the significance threshold… but it can be entirely by chance)

32 John August 26, 2015 at 10:55 am

Right. Even if the system now is “too conservative” the incentive is that pharmaceutical companies get proof of efficacy ASAP. With more rapid access to markets, we’d probably have to wait for 10 or 20 year reviews on efficacy.

33 Floccina August 26, 2015 at 11:45 am

@Moreno Klaus how much more cautious do you want them to be? Keep in mind that fraud is fraud. If MRK introduces a drug that they know does not work that is fraud. Of course Governments seem uninterested in prosecution fraud.

34 mavery August 26, 2015 at 12:13 pm

It’s not fraud to say that “clinical trials have shown that Drug X helps with Condition Y” if the trial results turn out to be incorrect. If a company has additional data showing the drug doesn’t work that they choose not to share, that’s a different matter and unrelated to the discussion above.

35 Doug August 26, 2015 at 1:51 pm
36 Nock August 26, 2015 at 3:23 pm

JPANDs is published by a bunch of quacks at AAPS (e.g. abortions cause breast cancer and leprosy comes from immigrants). Take their writings with a huge rock of salt. Have anything from JAMA?

37 jdm August 26, 2015 at 6:33 pm

The beta-blocker article says they were approved in 1981 – 34 years ago. Is there anything more recent?

I’m frankly somewhat disappointed that the author of this blog post hasn’t shared a few actual examples of relative recent (~last decade or so) Type-II errors and/or undue delays. It seems somewhat implausible that all (or even most) Type-II errors are undetectable.

38 dearieme August 26, 2015 at 8:37 am

“Conservative” and “aggressive” seem to me to be lousy adjectives for the problem under discussion. “Cautious” and “daring” might do better.

39 Floccina August 26, 2015 at 11:46 am

+1

40 Stuart Buck August 26, 2015 at 8:41 am

Recent headline from Forbes: “The FDA Is Basically Approving Everything. Here’s The Data To Prove It.” forbes.com/sites/matthewherper/2015/08/20/the-fda-is-basically-approving-everything-heres-the-data-to-prove-it/

I suspect that the real problem is that we’ve already found most of the low-hanging fruit in drug development, and it’s really hard to come up with a drug that makes more than a marginal and barely detectable difference for things like cancer. On the rare occasion that a drug is really effective, the FDA drops all barriers to rush it to approval ASAP (see Gleevec).

41 Art Deco August 26, 2015 at 12:17 pm

and it’s really hard to come up with a drug that makes more than a marginal and barely detectable difference for things like cancer.

???
Brain tumors, bone cancer, lung cancer, esophageal cancer, liver cancer, pancreatic cancer, ovarian cancer, and acute myeloid leukemia are, most of the time, killers. They account collectively for about a quarter of all diagnoses. Most people survive the rest, though there are subtypes and subtypes of subtypes which are virulent. There have been huge strides in cancer treatment in the last 50 years.

Your real problem is development of new antibiotics.

42 gwern August 26, 2015 at 5:03 pm

> Recent headline from Forbes: “The FDA Is Basically Approving Everything. Here’s The Data To Prove It.”

> BioMedTracker doesn’t just track what happens to NMEs, or new drugs. It follows closely every time a company asks the FDA for an approval…More than that, it has instituted new procedures to make sure it communicates well with drug companies before they file new drug applications.

So the rate of drugs from Phase I to approval, the total rate, may not have changed at all.

43 Jim Nazium August 26, 2015 at 8:52 am

The authors consider the cost of approving ineffective therapies, but do they consider the cost of approving therapies (effective or not) with serious adverse side effects? That seems to be an important part of the issue.

44 John August 26, 2015 at 10:57 am

Recall that among Dr. Fata’s crimes was pulling patients out of hospice for costly, ineffective, and painful treatments.

45 gwern August 26, 2015 at 5:30 pm

(If you have questions about a paper, it really does make sense to just read it, and not assume the authors are idiots.) Side-effects are handled on pg13:

> In this section, the two cost parameters, c 1 and c 2 , associated with adverse effects of medical
treatment and severity of the disease to be treated, respectively, are estimated…To estimate c_1 , which is the current cost of adverse effects of medical treatment per patient, we insert the corresponding numbers for the adverse effect of medical treatment in the U.S. from the U.S. Burden of Disease Study 2010 [18] into (6), and the result is c_1 = 0.07. The value of c_1 can be made more precise and tailored to the drug candidate being tested if the information from earlier clinical phases, e.g., Phase I and Phase II, is used. However, for simplicity, we only consider a common value for c_1 for all diseases.

If I’m following it right, this is an estimate that on average the side-effects from treatment are 7% as bad as the disease itself, The average level of side-effects per disease is drawn from

> [18] C. J. L. Murray, J. Abraham, M. K. Ali, and et al. The state of U.S. health, 1990–2010: Burden of diseases, injuries, and risk factors. JAMA, 310(6):591–608, Jul. 2013. https://jama.jamanetwork.com/article.aspx?articleid=1710486

The exact estimates for medical side-effects seems to be in the supplemental material giving the full table of all reasons for injury: https://jama.jamanetwork.com/data/Journals/JAMA/927436/JOI130037supp1_prod.pdf The descriptive term is “medical treatment”, with the estimates on pg157, where the columns are 1990 all-deaths, 2010 all-deaths, 1990 age-standardized death rate per 100,000, and 2010 age-standardized death rate per 100,000 (73, 127, and 26/32 respectively).

These numbers are from the Global Burden of Disease Survey, which itself synthesizes a bunch of the available epidemiological research on deaths from particular causes:

>

The GBD 2010, a collaborative effort involving 488 scientists from 50 countries, quantified health loss from 291 diseases and injuries, 1160 clinical sequelae of these diseases and injuries, and 67 risk factors or clusters of risk factors for 187 countries from 1990 to 2010. The overall aim of the GBD 2010 was to synthesize the world’s knowledge of descriptive epidemiology to facilitate comparisons across problems, over time, and across countries. Methods and summary results from the GBD 2010 for the world and 21 regions have been published.3,19,25- 31 Several studies focusing on results for a specific disease or risk factor have also been published or are in preparation.32- 34 Because the GBD 2010 uses a standardized approach for 187 countries, the results can be used to benchmark population health outcomes across different groups of nations. National burden of disease studies including a benchmarking component using the GBD 2010 have been completed for the United Kingdom32 and China.35 Details on the data, approaches to enhancing data quality and comparability, and statistical modeling and metrics for the GBD 2010 are published elsewhere.3,19,25- 27,29- 31

What GBD 2010 used to estimate adverse effects of medical treatment, I can’t immediately find out (it’s a big project) but it sounds as if they’d be using all the usual epidemiological studies one might cite about the high rate of adverse treatment effects, so I don’t consider this a particularly questionable part of the analysis.

46 Marge I. Nal August 27, 2015 at 4:59 pm

Seems like if you have questions about the paper *and you don’t have time to read it* then posting that statement and getting someone else *who both has time and is willing to read it* explain it to you makes a hell of a lot of sense. In fact, it worked for him. So your snark backfired, IMHO. -1

That said, that’s awesome of you to read it and comment honestly and intelligently on your findings. Good show. +2!

47 Ed August 26, 2015 at 9:00 am

Some of the discussion in the blog posts and the comments revolves around the difficulty of identifying when the FDA makes a “Type 2” error (delays/ denies approval of a drug it should approve).

One possible way to track this would be to compile data of drugs that the FDA does not approve, but other countries (Canada, Mexico, the Philippines, China etc.) approve. If there are a lot of these errors, you should be seeing effective, reasonably safe drugs available outside the US but not in the US.

I realize this is mixed up with lots of other issues, but its a start. There is a chance that the equivalents of the FDA in countries outside the US are either even more conservative than the FDA, or just follow whatever it is the FDA does. But if this is the case, it is an interesting finding in itself andit would be worth finding out why.

48 BenK August 26, 2015 at 9:06 am

The Forbes article is pretty silly. Yes, the FDA approves most things that make it to the final stage after lengthy discussions and all sorts of detailed reviews of each kind of clinical trial, and so on. The whole process is a ‘fail faster’ strategy so that a surprise failure at the end is unlikely. Surprise failures that late tend to crush companies and eat FDA reviewer time. Nobody wants that. So, yes, basically everyone who makes it to the Olympics finishes the marathon.

The real question is whether the FDA should be encouraging more companies to fail during post market surveillance (PMS; no joke) so that it can relax Phase I, II, even III. What if safety seems to be problematic in mice but might not be an issue in humans or might provide a balancing benefit? Should we encourage companies to proceed based on sketchy pre-clinical safety data with the hope of rescuing during efficacy and a bit of a trade-off for individual patients during PMS?

The people who claim we have discovered everything that works (low hanging fruit, sometimes the term) are basically wrong. If you say that we have discovered most of the easy things that are relatively inexpensive to manufacture, universally treat a common condition, work in a petri dish, mice, primates, and certain populations of humans amenable to testing… then maybe. But by relaxing several of those conditions, there are many efficacious medications to be discovered.

49 David C August 26, 2015 at 9:21 am

I agree wholeheartedly that the FDA is too conservative. It is politics that makes them so. In addition to the risk of approving an ineffective drug, there is also the risk of approving an effective drug with unrecognized side effects. This, more than anything else, is driving the size of Phase III trials. The FDA got hammered over the Vioxx incidence, and basically no amount of testing in arthritis trials was ever going to prove the link to heart attacks. Those trials couldn’t ethically compare Vioxx to placebo. It was only when Merck tried to extend the label into cancer prevention that there were large placebo-controlled trials, and the data on heart attacks became clear.

I think we, as a society, don’t have a consensus about how we feel about very effective drugs that have rare but serious side-effects. Is this let the buyer beware? I lean that way, but many do not.

I also have much more concerns about companies pushing ineffective drugs. They are happy to do it. I worked on development of a topical NSAID, and the sales and marketing folks were perfectly happy selling placebo effect.

Your comment that Type I errors correct themselves over time is certainly false. There are MANY therapies that have been shown to be ineffective (or even harmful) (think atheroscopic knee surgery or cortisone injections into the spine), and yet are used hundreds of thousands of times each year.

50 mavery August 26, 2015 at 9:33 am

The real question is whether the nominal Type 1 and Type 2 error rates the FDA purports to use are indicative of demonstrable Type 1 and Type 2 error rates. This question is related to findings about how “nearly everything published in peer review journals is wrong”.

So while the model the FDA uses may be very flawed, an overly-conservative model my be correcting (possibly under-correcting) for a system in which p-values are easily gamed.

51 Curt F. August 26, 2015 at 9:54 am

I would like to complain again about the ridiculous terms “Type I error” and “Type II error”. These are among the worst neologisms that science hath ever wrought. See Matt’s comments to a year-old MR post for more discussion of why these terms are regrettable. Can’t we all agree to use “false positive” and “false negative”?

52 TallDave August 26, 2015 at 9:56 am

Incentives matter.

It’s funny how many modern problem still come down to some flavor of Bastiat’s hoary old “That Which Is Not Seen.”

53 Moreno Klaus August 26, 2015 at 10:45 am

if the threshold for significance is higher for highly deadly diseases then…the market will become flooded with lower-quality drugs for terminally ill patients -> a few years later huge lawsuit.

54 Moreno Klaus August 26, 2015 at 10:46 am

*i dont mean terminally ill but patients which carry these potentially deadly diseases

55 Stuart Buck August 26, 2015 at 10:42 am

The paper cited in the post seems quite detached from the empirical evidence on FDA approvals.

Consider all of the evidence in the JAMA study described here: http://www.forbes.com/sites/harlankrumholz/2014/02/18/fda-drug-approval-policy-and-informing-patient-choice/

In the words of Yale’s Harlan Krumholz, “Many people may think that FDA approval decisions are uniformly based on strong evidence. We believe that patients and the public will benefit from being more fully informed . . . For instance, one-third of new drugs were approved on the basis of a single key study, which means that the findings were not replicated in a second study before approval. Also, for common diseases, the average number of patients in all of the studies supporting a new drug approval was just a bit more than 500, but one-quarter had fewer than 200 patients whereas another quarter had more than 1000. Smaller studies with fewer patients are generally less reliable than larger studies. Similarly, among new drugs approved to treat chronic conditions, such as diabetes or heart disease, fewer than half were approved on the basis of at least one study of patients that lasted 6 months or longer, and only 12% a year or longer. Yet these are drugs that patients will take over the remainder of their lifetime, requiring patients and physicians to extrapolate about long term risks and benefits.”

Consider also the review of FDA approvals for medical devices: http://medcitynews.com/2015/08/21st-century-cures-jama/

56 Effem August 26, 2015 at 10:42 am

Feels too circular to study. If the FDA were less conservative there’d be a higher incentive to put less-promising therapies up for review; which would have had the effect of raising the costs of being less conservative. I don’t see how you can isolate the variables required.

57 Axa August 26, 2015 at 10:44 am

FDA too conservative?

What a diversity of opinions on the FDA. If you ask some people that may benefit from a drug already approved in Europe they’ll say FDA is too conservative. If you ask other people they’ll say FDA is too permissive because X drug is common in the US but banned in the EU because of risks. Other people will say US independence as a nation is in jeopardy if the FDA fast-tracks a drug already in use in Europe. The FDA has to survive in this environment, and evaluate new drugs.

58 Stuart Buck August 26, 2015 at 10:46 am

Moreover, if the FDA is truly too “conservative,” then the drugs that DO get approved should be amazingly effective. By definition, setting a high bar would mean that the ones that get approved have indeed met a high bar.

Yet since 2002, the median survival benefit from ALL cancer drugs approved by the FDA is only 2.1 months:
https://twitter.com/jsross119/status/495906809694015488

I’d suggest that this isn’t a very high bar to clear at all, and that if not enough cancer drugs are getting through, it is much more likely to be because the drugs are utter failures than because the FDA is being mean.

59 John August 26, 2015 at 12:20 pm

That’s pretty bad.

60 Art Deco August 26, 2015 at 12:22 pm

You’re quoting a tweet on this subject??

While we’re at it, chemotherapy is the primary treatment for leukemias and lymphomas. For other cancer types, they’re meant to inhibit cancer recurrence or contain metastatic cancers for a time. The whole point is incremental improvement.

61 John August 26, 2015 at 12:44 pm

If you are right that chemotherapy is effective in many cases, and he’s right that the average benefit is 2.1 months, then obviously resources are misapplied.

62 Art Deco August 26, 2015 at 1:20 pm

He said ‘median benefit’. That excludes outliers. The outliers would be those ailments for which chemotherapy treatments are salient and those cases where they are decisive. While we’re at it, add up those increments over seven or eight decades and you get some impressive cumulative results. Only the most virulent cancers (e.g. pancreatic cancer) have survival rates which would have been about normal in 1925. As recently as 1965, a diagnosis of leukemia or lymphoma was a death sentence. Nowadays, only the acute myeloid leukemia is a death sentence most times, and you have a better than even shot if you’re under 50.

And, pace Ivan Ilich, treatments are decisive. Lung cancer and melanoma are just about the only major types for which changes in personal habits would be salient.

63 John August 26, 2015 at 1:36 pm

I haven’t read the JAMA article, but I worry still about what this “median” means.

64 Moreno Klaus August 26, 2015 at 2:57 pm

John: Many people will derive no benefit at all, from the drug. Thats why the number is so low. Also let’s not forget most of these diseases affect older people, and in some cases even if the drug is effective, the survival gain could be limited.

65 Art Deco August 26, 2015 at 3:42 pm

The median age at diagnosis for most cancers is about 66 years +/- 5 years. The life expectancy of someone that age is about 18 years in this country, so you’ve got some time left.

66 Stuart Buck August 26, 2015 at 1:10 pm

If you click on the tweet, you’ll see a Yale professor discussing the relevant chart with a link to a scholarly article from the JAMA network.

67 Art Deco August 26, 2015 at 1:21 pm

I’ll see 140 characters.

68 Stuart Buck August 26, 2015 at 2:27 pm

The nice thing about links is that they lead to further information elsewhere.

69 gwern August 26, 2015 at 3:16 pm

> Moreover, if the FDA is truly too “conservative,” then the drugs that DO get approved should be amazingly effective

*Or*, besides the effect size being so large that even small studies showed it, it could just mean that the cancer drugs got large enough trials to detect the effects or some of them got lucky despite low power or the FDA implicitly raised acceptable p-values because cancer drugs are a priority. Remember that NHST has a 4-way tradeoff: power vs p-value vs n vs effect-size. Any mixture can be chosen.

70 Stuart Buck August 26, 2015 at 8:11 pm

Exactly. Thus, if the regular old p-value that the FDA demands is somehow too strict, that can only be because the effects seen in typical trials are so small that you need large N’s in order to detect them, and you still might not have enough power to detect the (vanishingly small) effect at all.

But I think the real problem isn’t that the p-value threshold is too strict, but that no one is coming up with drugs that have meaningful effects. Again, whenever someone comes up with a drug that is actually effective (Gleevec), no one thinks that the p-value threshold is hard to meet, and no one thinks that the FDA is standing in the way (it instead bends over backward to rush a drug like that through as quickly as possible).

71 gwern August 27, 2015 at 2:44 pm

‘Vanishingly small effect’ is just a rhetorical flourish. An effect can be very substantial on a population-scale while not satisfying a particular arbitrary p-value or power level at affordable ns . That’s kind of the whole point of the OP paper!

72 Stuart Buck August 27, 2015 at 8:50 pm

Vanishingly small is a term I’m using to mean “even less than the 2.1 months of survival that can obviously get FDA approval.” If a treatment offered 1 week of survival, on average, then if you multiply it by, say, 100,000 patients, it looks like a large effect, but it is still pretty meaningless in the context of any actual patient, so it is a little overly emotional for the original post to refer to Stage IV pancreatic cancer patients hypothetically* being denied the right to use an “effective” treatment. And to the extent such patients are worried about every week of survival, they might prefer to be sure that new treatments offer at least *some* benefit, as opposed to having a margin of error that includes killing them 6 months early, for example.

I’m not even saying I disagree with the original paper, in the end. It’s just odd to have a discussion like this, in seeming obliviousness to how low a bar the FDA actually sets in many cases.

* No one has cited any _actual_ examples of this happening, one might note.

73 Edgar August 26, 2015 at 11:08 am

“… and the New York Times writes stories about them…” Do we have trouble in paradise? Has Alex grown envious of Tyler’s New York Time’s gig? Is Tyler’s celebrity casting a shadow? Will the revolution splinter into factions? Tune in tomorrow for another thrilling episode of As the Marginal Revolution Turns!

74 prior_approval August 26, 2015 at 12:42 pm

Small steps to a much better soap opera….

75 ThomasH August 26, 2015 at 12:21 pm

The “too conservative” finding seems plausible but how does that jibe with the drugs that are approved that “effectively and “safely” postpone death by epsilon months?

76 BurningFrog August 26, 2015 at 1:35 pm

1. This issue really needs more facts to be communicable to the public. This study is a good step, but until we have body counts I don’t think much will move.

If you could say – and back up with research – that the FDA process leads to 57.000 more deaths per year than an alternate process, that could actually sway opinions.

2. Has anyone compared the policies of other major countries? Are there drugs that work well in Europe/Canada/China that are banned by the FDA?

77 Black Death August 26, 2015 at 2:52 pm

I’m surprised that nobody has mentioned thalidomide, which was a drug developed in Germany and first marketed there in 1957. It was used for anxiety, insomnia, and unfortunately, for morning sickness of pregnency. In the subsequent years, over 6,000 children were born there with missing or absent limbs (about 10,000 worldwide). Many other birth defects were traced to the drug, which was sold over the counter. About half of these children died.

Thalidomide was sold in Canada but not in the US. The drug was not approved in the US due primarily to the efforts of Frances Kelsey, MD, Ph.D, an FDA scientist who had serious doubts about the adequacy of the documentation of the drug’s safety. Dr. Kelsey’s efforts largely spared the US from the epidemic of thalidomide babies that occurred elsewhere. President Kennedy gave her a medal. Dr. Kelsey died earlier this month at the age of 101.

More than fifty years later, the shadow of the narrowly averted thalidomide disaster still haunts the FDA drug approval process.

78 Randall Parker August 29, 2015 at 11:55 pm

I would hope that thalidomide fears play no part in cancer drug approval. After all, existing cancer treatments are highly toxic already.

Since many cancers kill within months or a few years a real cure is going to provide a really clear signal. Detecting efficacy is much harder for treatments that have disease prevention as a goal (like reducing the risk of getting cancer or heart disease in the first place).

79 Jeff August 26, 2015 at 4:41 pm

There is yet another way in which the FDA is too conservative. Sometimes new uses are discovered for older drugs that have gone out of patent. Low-dose ketamine, for example, appears to be quite effective for depression and it acts much faster than the usual antidepressants. We know this from several small studies. But since ketamine is out of patent, no private party (drug company) is going to spend the millions of dollars it would take to run Phase I, II and III trials to get ketamine approved for this use. While doctors are free to prescribe ketamine for depression as an “off-label” use, insurance companies don’t have to pay for it as they would if this use were FDA-approved. The result is that adoption of the drug for this use is very slow, with no one having much of a financial incentive to push it along.

When there’s no one with deep pockets willing to pay for all those trials, the FDA should make it easier for small groups of doctors and patients to get drugs approved without spending enormous amounts of money.

80 Art Deco August 26, 2015 at 9:34 pm

Why should insurance companies be paying for a tranquilizer which can be produced and distributed by generic manufacturers for modest prices? Why should they be paying for psychotropics at all for patients who are not on anti-psychotics?

81 Jon August 26, 2015 at 9:53 pm

Does the FDA use a fixed 2.5% confidence interval? In many cases the choice is not between death from the disease and being cured, but a matter of whether the survival time is lengthened.

Sorry Alex, the piece you cite is not a technical proof that the FDA is too conservative or has a net negative impact; it is just a proposal of a a framework for determining approvals.

82 Boonton. August 26, 2015 at 10:04 pm

For example, the standard Type I error of 2.5% is too conservative for clinical trials of therapies for pancreatic cancer

This seems to imply if we look at the list of rejected drugs we’ll find something good for pancreatic cancer that failed because it scored 2.75%. But with almost all drugs submitted to the FDA getting approved it doesn’t seem likely there would be many drugs that fit this category.

Likewise a drug with a 2.75% rate could be sold in a foreign country whose FDA was not as conservative. That would provide an opportunity to acquire more data to establish the drug’s usefulness. Yet we never hear about anything like that happening?

Finally isn’t FDA approval done by vote? This would suggest that if you had some drug on the border line a case could still be made for it.

83 Jon August 27, 2015 at 10:34 am

I don’t believe that the FDA rigorously uses the 2.5% anyhow. I have seen some reports that indicates in many situations the drug company only has to demonstrate that the new medicine is not inferior to alternatives.

84 Mac Choux August 27, 2015 at 8:22 pm
85 K August 29, 2015 at 9:28 am

Alex,

From the conclusion of the paper:

“For example, for pancreatic cancer, if the prior probability of efficacy is 60% instead of 50%, the size of the BDA-optimal test would be 51.2% rather than 27.9%, leading to many more approvals of such therapies.”

It’s an interesting assumption that if thresholds were raised, that the prior probabilities of efficacy wouldn’t change. If in fact the significance threshold for pancreatic cancer drugs were set at 27.9% instead of 2.5%, you would only need to run around 2 trials on average, to have a 50% chance of getting clinical approval to treat patients with any random ineffective “drug”. The economic incentives, and therefore the prior probabilities of a drug working, are directly linked to the significance thresholds.

Rather than the FDA being “dramatically too conservative”, I’d say it’s the authors of the paper who are being comically naive. Do they really think that if the significance threshold for a drug for a deadly illness were 51.2%, that there would be any reason for the drug developer to care what they are submitting for trial? I predict a surge of FDA approved homeopathics.

86 jane September 1, 2015 at 10:46 am

And yet Forbes just reported that the FDA has most recently approved about 90% of applications to market new drugs. Your argument relies on two questionable value judgements: first, that people with fatal illnesses should wish to devote their remaining time to scrabbling desperately after even the slightest mirage of a cure, at no matter what cost, and secondly, that the interests of corporations should be weighed along with the interests of patients. You say that approving bad drugs “at least has the advantage of helping to incentivize R&D” – well, first of all, I do not want to be the one who dies in an ICU of a side effect for the sake of incentivizing R&D.

But secondly, and perhaps more importantly, big-business libertarians like to say “If you subsidize something, you get more of it.” Rewarding drug companies for inventing and marketing drugs that mostly affect surrogate measures at great cost in side effects “incentivizes” them to produce even more drugs that mostly affect surrogate measures at great cost in side effects. To do that, after all, is much easier than to produce a drug that substantially changes the long-term course of a chronic illness.

87 Clay B September 16, 2015 at 1:34 pm

Maybe they should have more than one level of approval…like Certified (thouroughly tested) and Experimental. Or they could rank drugs on a 1-10 for safety and effectiveness.

Comments on this entry are closed.

Previous post:

Next post: