Rating the FDA by Division

by on April 24, 2014 at 7:28 am in Economics, Law, Medicine | Permalink

In previous work, I have argued that asymmetric incentives make the FDA too risk averse with the result being excessive drug lag and drug loss. The FDA, however, is not a monolithic agency, it is divided into divisions which oversee different types of drugs. The divisions have different cultures, expectations histories and understandings. In my latest paper, written with Tufts researchers Joe DiMasi and Chris Milne, we put aside the question of global efficiency and ask a different question. How do the FDA divisions rate against one another? What we find is quite surprising: some of the FDA divisions appear to be much more productive than others. From the abstract:

After reviewing nearly 200 products accounting for 80 percent of new drug and biologic launches from 2004 to 2012, the authors find wide variation in division performance. In fact, the most productive divisions (Oncology and Antivirals) approve new drugs roughly twice as fast as the CDER average and three times faster than the least efficient divisions—without the benefit of greater resources, reduced complexity of task, or reduction in safety. The authors estimate that a modest narrowing of the CDER divisional productivity gap would reduce drug costs by nearly $900 million annually. The worth to patients, however, would be far greater if the agency could accelerate access to an additional generation of (about 25) drugs. Greater agency efficiency would be worth about $4 trillion in value to patients, from enhanced U.S. life expectancy. To reap such gains, this study encourages Congress and the FDA to more closely evaluate the agency’s most efficient drug review divisions, and apply the lessons learned across CDER. We also propose a number of reforms that the FDA and Congress should consider to improve efficiency, transparency, and consistency at the divisional level.

Andrew von Eschenbach a former Commissioner of the FDA and Director of the National Cancer Institute and now chairman of the Manhattan Institute’s Project FDA wrote a foreword to our paper. Eschenbach writes:

The authors of this report have taken a giant step…by assembling and analyzing a wide array of publicly available information about the relative performance of individual CDER divisions….Continuous, quality improvement measures routinely used by private industry could serve FDA leadership, sponsors, and patients by discerning factors that contribute to an optimal level of performance and, more important, disseminating such practices to ensure that all divisions achieve that performance. The payoff for such an effort could be enormous.

…Process improvement should not be a controversial proposal. An organization like the FDA—which is over a century old and which has maintained its current, basic organizational framework for decades—requires new tools to adapt to changing circumstances.

…I have enjoyed no greater privilege in my professional career than serving alongside the FDA’s talented staff. Today, the agency has more potential than ever to help the U.S. lead the world in advancing a biomedical revolution, one that will have an impact on every aspect of America’s economy and health-care system by improving health, increasing productivity, and reducing overall health-care costs.

…this report should be viewed as a positive, constructive contribution to a desperately needed dialogue on how to assist the agency in fulfilling this vital national goal.

1 david April 24, 2014 at 7:46 am

It is hard to take a proposal for reform in good faith when the authors keep whispering “abolish it, abolish it”.

2 Cliff April 24, 2014 at 11:22 am

Why? Is there some way that increasing efficiency might somehow result in it being abolished?

3 Rahul April 24, 2014 at 8:01 am

Could the Oncology division’s behavior be explained given the more mortal baseline scenario? Could that partially explain a higher risk appetite or more aggressive approval attitude?

A relatively healthy patient dropping dead because he went on a heartburn drug draws more attention than a small spike in what’s already a high mortality game.

4 Mark Thorson April 24, 2014 at 11:06 am

I’d guess that political pressure has something to do with it. There’s an endless supply of deperate cancer patients and their families. Recently, a bunch of them have been lobbying Congress for increased access to a putative anti-cancer drug promoted by a certain quack doctor in Texas, and FDA partially caved in by allowing access with restrictions.

The champions of lobbying are the AIDS activists, of course. They’ve achieved a lot, with regard to accelerated approval of drugs.

5 Roy April 24, 2014 at 3:14 pm

Which would explain the antivirals process as well

6 andrew' April 25, 2014 at 5:15 am

And now we have drugs for aids. The safe and effective bar moves based on the prognosis of the patient.

Remember, it is not the fault of cancer patients that we have to do human experimentation on them and have to ram it through an unacceptable regalatory and financing system that isn’t mature enough to handle experimentation and known treatments in the same process.

7 Doug April 24, 2014 at 12:11 pm

It might also have something to do with the culture of oncology in general. Since the field pretty much just consists of pumping the patient full of poison until the disease goes away, they might not have that much of an innate worry about side effects.

8 Rahul April 24, 2014 at 12:15 pm

Indeed. The side effects of Chemo / Radiation set a very high bar. Anything that doesn’t pretty much kill you is an acceptable side effect.

9 Jan April 24, 2014 at 8:44 am

Those researchers that Alex collaborated with are based at Tufts Center for the Study of Drug Development. This organization receives nearly half its funding in unrestricted grants from the pharmaceutical industry. They know what they are paying for. Their work tends to be heavily biased. http://csdd.tufts.edu/about/financial_disclosure

10 Dave Anthony April 24, 2014 at 9:48 am

It’s clear that you meant their funding from pharma to be a pejorative, but It is also not obvious that the pharmaceutical industries interests and biases are entirely un-aligned with the public. So this study shouldn’t be discounted just because of “guilt by association”, which is really just a lazy form of thinking.

11 dan1111 April 24, 2014 at 9:59 am


12 Rahul April 24, 2014 at 10:07 am

It’s still good to know about funding sources. I guess that’s why many journals insist on comprehensive Conflict of Interest Statements from all authors. Seems fairly common in Pharma Journals.

As an aside, is there a peer-reviewed version of Alex’s paper?

13 Jan April 24, 2014 at 2:23 pm

Not guilt by association, guilt by rigorously demonstrated bias. Industry funding biases research in various ways. Here is a summary of some of the studies on how industry $ impacts research findings: http://www.scientificamerican.com/article/trial-sans-error-how-pharma-funded-research-cherry-picks-positive-results/

“It is not obvious that the pharmaceutical industries interests and biases are entirely un-aligned with the public.” I’m not sure about that. It is the fiduciary duty of the board of any publicly traded company to make money for the shareholders, and as much of it as possible. That’s it.

14 Curt F. April 24, 2014 at 2:35 pm

Bias is not the same as falsity. Plus, psychological research is pretty compelling that *everyone*, not just big corporations, suffers from bias. Who are these angels that do unbiased research?

15 Jan April 24, 2014 at 3:37 pm

Oh sure, bias is not the same as falsity, but it sets the stage for it.

A researcher that receives an NIH grant for a study may suffer from their his own personal biases, but that is true for any researcher, no matter who the funder is. It is not clear to me what additional bias is introduced by receiving funds from a government agency, whereas with industry funded research on commercial products the presence and direction of the bias is quite obvious.

16 @Jan. April 24, 2014 at 4:11 pm

Awhile back a fairly ridiculous study of the effects of marijuana on the brain was published. See this blog post (http://liorpachter.wordpress.com/2014/04/17/does-researching-casual-marijuana-use-cause-brain-abnormalities/) for a discussion. You will note that a spokesman for the government funding agency is quoted as saying “As the National Institute on Drug Abuse, our focus is primarily on the negative consequences of marijuana use,” said Shirley Simson, a spokeswoman for the drug abuse institute, known as NIDA. “We generally do not fund research focused on the potential beneficial medical effects of marijuana.”

So for that particular government agency, they seem to want to know the results ahead of time, then maybe the will fund the study.

Examples like that confuse me. Why do you expect the bias of government funding agencies to be any better or worse than the bias of private companies?

17 Jan April 24, 2014 at 5:03 pm

Most federal agencies don’t have any research agenda other than to contribute to the scientific body of evidence in ways that benefit society. Also, once a federal agency awards grant money, that is pretty much the end of their control — the researchers are free to publish their findings, which must often be made free and available to the public. They have much less ability to influence the conclusions and publishing than industry/privately funded research.

As to your concern about NIDA specifically, here is their portfolio of research on the therapeutic benefits of marijuana. I’m not sure an agency with a harmfully narrow mindset and aversion to genuine research on all aspects of drugs would go down this road. http://www.drugabuse.gov/drugs-abuse/marijuana/nida-research-therapeutic-benefits-cannabis-cannabinoids

18 andrew' April 25, 2014 at 7:06 am

Jan, you are wrong. I mean that definitionally. Just because you want federal agencies to be only for the public good (whatever that means to you) doesn’t make that true.

Alex just explained one way why it’s not true.

19 andrew' April 25, 2014 at 7:08 am

Unless you think some fda divisions are more biased than others…but even that falsifies your assumption.

20 Jan April 25, 2014 at 8:23 am

I just showed you why I am right. With data and anecdotes and logic. Please refute those.

21 Andrew' April 25, 2014 at 2:21 pm

You made an assertion and that’s all and I have probably a hundred times just on this blog refuted the assertion.

You can’t see what bias there is other than individual researchers?

Well, what about the part where the committee of peers of…researchers decides funding? There is one.

And this is a system set up for the express purpose of…reducing bias! News flash: it’s not perfect at fixing the problem that makes it necessary. It is just better than some of the alternatives.

Now, imagine if they didn’t toss the money over the fence to semi-independent researchers to decide where the money goes? IT WOULD BE MORE BIASED.

22 Jan April 25, 2014 at 3:28 pm

Of course there is bias by more than individual researchers. That is why it is bad for industry to fund studies they have a stake in the outcomes of.

What you talk about helps a little, doesn’t end the bias.

23 Michael April 24, 2014 at 11:42 pm

Zero skepticism about research results based on funding links: bad.

Total skepticism about all research results based on funding links: worse.

One-sided skepticism about any research you don’t like because of funding links: worst of all.

24 prior_approval April 25, 2014 at 2:29 pm

‘but It is also not obvious that the pharmaceutical industries interests and biases are entirely un-aligned with the public’

So, that study that Tamiflu is about as effective as placebo didn’t get big play in the news in the U.S.? In comparison to Germany, for example, where it was held up as an example of how easily governments are swindled. – http://www.cochrane.org/features/tamiflu-and-relenza-getting-full-evidence-picture

This including the reporting that the only study that provided evidence of efficacy was done by Roche employed researchers?

25 andrew' April 25, 2014 at 6:35 am

Why are people so worried about alledged industry bias?

Seriously. Where is the actual evidence for this that it is the big problem.

As far as I know, though i havent been keeping up with the news, the big problem is we are all still going to die.

I think vaccines are a fiasco and it never occurs to me, even in the case of obvious recent lobbying problems with HPV that the producers are the main problem.

Where is my Arnold kling bat symbol? Hey, who moved my kling signal?!?

26 prior_approval April 25, 2014 at 2:30 pm
27 Bill April 24, 2014 at 10:25 am

So, ah, you’re for increased funding for the FDA?

28 andrew' April 25, 2014 at 5:17 am

Ah here we see one of the biases of federal agencies.

Every failure demands more funding. That is an obvious conflict of interest.

29 Harry April 24, 2014 at 11:26 am

Why not establish a reciprocal approval program with regulatory authorities in peer OECD countries? Approval by one authority equals approval for all the countries in the group. We import other products that require regulatory approval and are just as central to health and safety (ranging from agricultural products to nuclear power plant components) and we export patients to medical facilities all over the world. What’s so special about pharmaceuticals and medical devices that FDA and only FDA approval counts?

30 Abe Froman April 24, 2014 at 11:51 am

Nice job Alex. This is important work.

31 ed April 24, 2014 at 12:29 pm

Very interesting.

Maybe you’d get more attention for this kind of important work if you framed it as a question of “health inequality.”

32 Nathan W April 24, 2014 at 3:10 pm

I cannot emphasize more that I disagree with speed of evaluation or number of appprovals as the criteria to evaluate productivity.

This could equally be interpreted as shoddy work and approval of too many drugs after too little investigation.

I do, however, strongly support the idea of evaluating it by division. As much as I argued against the proposed criteria, it would be one of many useful indicators. I would like to know the share of division spending allocated to attending industry-sponsored events, and whether this relates to other variables which suggest that maybe some divisions are excessively open to the perspectives of industry interests at a cost to the general public.

33 Thomas April 24, 2014 at 11:50 pm

Without reading the paper, the abstract quoted in this post addresses your concern:

“three times faster than the least efficient divisions—without the benefit of greater resources, reduced complexity of task, or reduction in safety”

34 andrew' April 25, 2014 at 5:30 am

Reciprocal approval would be a good start, but then will come the howls about approval shopping. How ’bout just (expanded) trial approval?

Different divisions are apples and oranges until the bottom line comes in, are the drugs in totality safe and effective relative to the other departments?

The fundamental problem is that you can’t put chemicals in binary buckets of being acceptable or not. Safe and effective is a charade. There are just effects we know and effects we don’t know.

Botulinum toxin, the most toxic substance known to man is also injected for purely cosmetic objectives and it is fine.

35 Michael April 25, 2014 at 12:11 am

“Most federal agencies don’t have any research agenda other than to contribute to the scientific body of evidence in ways that benefit society. Also, once a federal agency awards grant money, that is pretty much the end of their control — the researchers are free to publish their findings, which must often be made free and available to the public. They have much less ability to influence the conclusions and publishing than industry/privately funded research”

as Gary Taubes nicely illustrates in his book, “Good Calories, Bad Calories.” Scientists who disagreed with the accepted wisdom on the evils of fat in the diet were accused of being corrupted by industry grants even if they had received most of their money from government agencies that were looking — unsuccessfully — for evidence to back the fat-is-bad theory. Meanwhile, scientists who went along with the conventional wisdom on fat weren’t criticized for the corporate money they’d received from food companies.

Federal Agencies are as big a pusher as industry. If you say what the agency says, then it’s okay. If you say something that isn’t what the agency says, or that may be parallel to what industry says, that makes you suspect.

36 andrew' April 25, 2014 at 4:51 am

The best funding format is directed by committees of researchers. Even they are biased.

That fed agency’s major bias may be cya is…kind of what Alex said.

That drug testing funding is mostly private is…how it is.

37 su May 15, 2014 at 2:57 pm

Might there be characteristics of the various disease areas that contribute strongly to differences in approval time? For example, neurological and psychiatric diseases can be very difficult targets due to a relative lack of good biomarkers or endpoints for efficacy, and greater “fuzziness” in evaluation parameters might lead to uncertainty during review and thus longer approval times. This might be captured to some extent by the clinical development time variable, but it’s unclear how good a proxy that is for scientific complexity. Similarly, as another poster noted above, what’s considered safe, efficacious, and worth the risk may be vastly different in different disease areas.

Comments on this entry are closed.

Previous post:

Next post: