Rating the FDA by Division: Comparison with EMA

Reporters have been asking the FDA about my paper with DiMasi and Milne, AN FDA REPORT CARD. As you may recall, the upshot of that paper is that there is wide variance in the performance of FDA divisions. Here, for example, is the mean time to approval across divisions. MeanTimetoApproval

Our simple index, discussed in the paper, suggests that these differences are not easily explained by factors such as resources, complexity of task or differences in safety tradeoffs across divisions. In responding to our paper, however, the FDA has said that similar differences in time to approval by drug type are seen at other drug approval agencies. If true, that would be an important criticism.

Fortuitously, some relevant data crossed our desk recently. The Center for Innovation in Regulatory Science (CIRS), a UK based research consortium, compared median review times at the FDA to the next most important drug regulatory agency in the world, the European Medicines Agency (they also look at the Japanese agency). To their credit, the FDA is faster on average than the EMA (thanks PDUFA!). What is relevant for our purposes, however, is to compare differences across divisions.

The CIRS breaks drugs into broader classes than we used but the story they tell for the FDA is similar to ours; anti-cancer drugs, for example, are approved much more quickly than neurology drugs. The story for the EMA, however, is very different than for the FDA. For the EMA all types of drugs are approved in roughly the same amount of time.

FDA Relative to EMA

We have argued that the wide variance in performance at the FDA is suggestive of differences in productivity. The fact that we do not see the same wide variance in performance at the EMA is supportive of our argument. Our goal and conclusion still stand:

We support further study to identify the policies and procedures that are working in high-performing divisions, with the goal of finding ways to apply them in low-performing divisions, thereby improving review speed and efficiency.


"We support further study..": doesn't everyone, always? That's where livelihoods lie.

This the academic version of "have a nice day" or if you are the fda "I said good day!"

The goal is slightly different here as we want the FDA to do the further study and they have more data about FDA personnel and procedures than we do.

Oh very well. The court rules "Not proven".

By the by, is there any scientific reason why it should be axiomatic that all the different sorts of drug should take the same time to approve?

Most of the difference is easily explainable if you consider benefit/possible harm ratio.

To make it simple, cancer is fatal. Having a high cholesterol may kill you over time. Meanwhile, we've pretty much figured out how to treat cardio disease - but nowhere near there yet in most cancers.

Therefore, regulators have to be more careful about approving a new cardio drug - you need giant data sets to prove that your potential drug works and has few side effects in lots of people over long periods of time.

For cancer, it's much more of a shoe in. Fewer people take the drugs, and adverse effects don't matter as much

Other factor is ease of proof of efficacy.

Neurology is black box stuff, and hard to prove. Antiviral is mostly just proving lower levels of viral load.

Cancer I get...well I will if I live long enough. Antiviral is HIV, right? The rest are some odd buckets. Cardiovascular and Renal?

Maybe the relevant buckets should be based on patients' prognosis.

You assume good faith on the part of FDA researchers. What is the basis for this? Are there stakeholders inside and outside the department who benefit from lengthy delays? Might not the panel find that the quicker departments are moving ahead too quickly?

Nah. There are certain "scientist" that withhold their data from the public and in the event that someone unauthorized gets the data by accident you send a letter to said person threatening to sue them if they share the data and further claim the threatening letter they sent is copyrighted and the receiver is not allowed to share it with anyone else. And no I'm not talking about the NSA.

And here is a bit of history in terms of that British organization -

'The CMR International Institute for Regulatory Science was established in 2002 under the auspices of CMR International Ltd. to assume the important role of providing a neutral forum in which senior executives from pharmaceutical companies, government regulatory agencies and academia could discuss current issues in an open-minded, intimate, and interactive environment.


CIRS – The Centre for Innovation in Regulatory Science – is a neutral, independent UK-based subsidiary company, forming part of the Intellectual Property and Science business of Thomson Reuters.' http://cirsci.org/history

Which just happens to nicely dovetail with this slightly more extensive history -

'The Centre for Medicines Research (CMR) was created in 1981 as a division of the Association of the British Pharmaceutical Industry (ABPI).

CMR’s initial role was to challenge public policy and regulatory issues that could adversely affect drug development. Early success, for example in amending long-term study requirements, placed CMR at the centre of pharmaceutical analysis and reform.

CMR’s greatest early success was in steering regulatory change. By carefully monitoring regulatory review times throughout the world, and facilitating international collaboration forums for industry, regulatory authorities and academia, we helped to reform and harmonize regulatory guidelines throughout US, Europe and Japan.

In the 1990s, in partnership with the world’s leading Pharmaceutical R&D organizations, CMR pioneered the development of reliable approaches to benchmarking pharmaceutical R&D.

In 2002, CMR International became a fully-independent, private company that was acquired by Thomson Reuters in 2006.

In parallel, we continue to support and develop the independent, not-for-profit CMR International Institute for Regulatory Science. The Institute continues to provide thought leadership in the development and implementation of regulatory policy.' http://cmr.thomsonreuters.com/who/history/

O brave new world, that has such coincidences in't!

If it is measured, it will be managed. I would expect your work to lead to a significant reorganization of the FDA.

Well done.

As I've said before, the thalidomide "success" resulted from waiting for first movers to take on the risk and not an actual protocol. I wonder how much that "success" has influenced current custom.

Resulted partly to mostly from I should say I guess.

Does the same measure of efficiency apply to departments at GMU?

Have you taken over prior_approval's role as resident troll?

I don't understand how anyone even thinks that constant unrelated harping is even remotely witty or relevant.

Good for the goose, good for the gander.

One criticism I have is that from the comment: "To their credit, the FDA is faster on average than the EMA ", is that it's not clear if any departments of the FDA are lagging.

Perhaps, Nervous System research really does take much more time than Cancer research. However if the EMA bureaucratically sets a slow march milestone schedule so that every department can meet it,, then the results would be as reported. IE. consistent times, but a higher average.

What was the average number of days for review per division for the EMA?

Excellent point and a very good question. You can see the raw data here, on page 3:


The FDA is much faster to approve in every single drug category.

I think you might be right, but the linked report is in Median times whereas Alex's graph is in Mean times. So, the data isn't directly comparable. However, it does look as if this is less an issue with some FDA divisions and more of an issue with the EMA.

Also of note: PMDA exhibits a very similar pattern to the FDA as far as differences between drug categories. The FDA has a much greater variance than the other agencies, particularly on the downside (longer times)

If it is measured, it will be managed. I would expect your work to lead to a significant reorganization of the FDA. - See more at: http://marginalrevolution.com/marginalrevolution/2014/06/rating-the-fda-by-division-comparison-with-ema.html#sthash.gU6zKFyi.dpuf

If the FDA is faster on average than the EMA, why scale the second graph to each organizations fastest category? Is that a standard normalization strategy? To my untrained eye it just seems like a fox-news tactic to make the FDA's bars seem much bigger than the EMA's. What if you scaled to the slowest? Then the EMA just looks incredibly incompetent and the FDA looks like it saves time where it can. What if the EMAs response time in its fastest category is actually slower than the FDA's slowest? The scaled graph doesn't tell me this. I'm not saying there isn't a good argument discussion here, but this post seems a bit undersupported as is, am I totally off base here?

Yes, you are a little off base. Alex isn't interested in how the FDA compares to the EMA per se. He is interested in how the divisions within the FDA compare to one another and why they are so different. One hypothesis is that it's all due to what the divisions are doing. The comparison with the EMA and the work in the paper Alex wrote suggests that is not the case. If Alex is right that there are productivity differences between the divisions that suggests important room for improvement at the FDA to great potential benefit of patients.

Why not consider the approval time delays based on the amount of advertising and product promote spending that follows for each drug in the US and Europe.

A cancer or antiviral drug is not going to have $100 million annual marketing budgets like Viagra and Vioxx got after approval. The reason for the marketing budget is to deliver as customers, and essentially test subjects, millions of individuals who are far more diverse than the the test population that won approval. Simply calling for the 10,000 test subjects to be more diverse when the tests are based on among other things patient populations in other nations with different but homogeneous populations than the US.

One of the characteristics of "socialized medicine" is the selection of prescription of drugs is much more top down determined by cost-effectiveness. A new drug with marginal or no advantage over existing drugs is far less likely to be prescribed outside the US. Its use will be limited to the patients that the old drugs can not effectively help.

Why was Vioxx so heavily marketed? What was its advantage over existing drugs to the actual patients and society?

Only if you reject the principle that economic profit signalling economic inefficiency is a virtue can the marketing strategy for Vioxx be considered beneficial to society. Diverting money from other more productive labor to generate monopoly profits and stock price inflation does not lead higher rates of employment - more employment would involve cheaper alternatives to Vioxx that eliminated the monopoly profits. Cheaper alternatives already existed.

More Vioxx testing would have not merely delayed its approval, but also limited its use to only those cases where the existing cheaper drugs were not more cost effective. And Vioxx was pulled totally from the market for business reasons even though it was worth the cost and risks for a small population of patients.

Sorry, but Alex and several of the commenters here are not considering several relevant points in interpreting this data. I am a pharma industry professional who has worked at several large firms who carries no water for the FDA but I understand what they are supposed to be doing and respect it. The FDA and EMA's determination of whether to approve a drug is dependent on evaluating the costs and benefits of the treatment as reveled by clincial trial data. The construction of clinical trials is depenent on the cost/benefit bar that must be cleared for approval, which is quite different across these treatment categories. Cancer and infections are the lowest bars to clear as there are still several types of cancer with few or no effective treatment options, and both types of patients may be dying rapidly. Moderate or even severe side effects can be tolerated if the drug extends the patient's life significantly or has even a moderate likellihood of major improvement, relative to the alternative of no effective treatment. Many cancer drugs are very effective for a small subset popultion, but only somewhat or minimally effective for others, in ways that are as yet unpredictable. Similarly, the newest ant-infectives are typically only used when resistance to older drugs is observed, so they are used for the most dire cases where other treatments are unavailable. Cardiovascual drugs, on the other hand, will be taken by many people who are still fairly healthy in the hopes of lowering the chance of a heart attach or stroke (e.g. statins for high cholesterol). A severe side effect that shows up in even a tiny fraction of patients might cause the drug to be unapprovable or unprofitable, as potentially millions of patients will take drug, and there are several safe options already on the market for most cardiovascular conditions. The bar to be cleared for this type of drug is much higher and thus the clincial trials must involve many more patients to complete a thorough evaluation. The data is much more complicated, and the evaluation of costs vs. benefits to the patient population as a whole may be quite difficult relative to cancer drugs. As a consequence, the clinical trials must be performed on large populations to ensure that all side effects are revealed. Evaluation of nervous system drugs for depression or Alzheimers, may be complex for the reasons stated above, and also because the science of the nervous system is still poorly understood compared with say, anti-infectives, and may involve patients with a lot of comorbid conditions. Your data shows more or less what I would expect given these constraints. The EMA may show less variance in their review times across categories than the FDA, but there can be many reasons for this, and your analysis does not consider enough factors to prove that the FDA is doing their job inefficiently or ineffectively, all things considered.

Onyx Mousse, these are time from NDA submission to approval times. Nothing to do with clinical trial time. In fact, we test your theory in the paper by seeing if clinical trial time explains approval time. It doesn't. See the paper for more details.

The differeneces are due to political pressure.

While there may be other factors at play, as a pharmacologist I would say that the increase in time between divisions roughly parallels with the complexity and heterogeneity of each disease or body system targeted. A drug designed to treat a cancer or a virus has a specific, well-validated target and practically binary objectives—reduce the tumor size or viral population. These drug effects can be readily demonstrated in cell cultures and animal models using the human forms of cancers and viruses, so it possible to make confident predictions of clinical efficacy with a few well designed experiments. In a general sense designing a drug to treat either disease can be relatively straightforward when compared to those of other body systems since the disease target is known directly (the cancer or virus itself) and the drug effect is easy to measure.

On the contrary, a cardiovascular drug could be designed to target muscle cells of the heart, nerves involved in cardiovascular control, the muscles that constrict arteries, the cells of the liver that synthesize cholesterol or the proteins that carry it in the blood, etc., in addition to all of the different hormones that regulate those actions. So if you wanted to design a drug you would have to identify the specific way in which your drug would affect a specific pathway and how that modulation would affect a potential disease state, such as a hear attack, which themselves are very difficult (if not impossible) to induce in an animal model, let alone a cell culture. So not only would you have to convince the FDA that your drug would work without an effective preclinical measurement of the actually disease you’re fighting, but you would also have to convince them that the mechanism you’re targeting is directly implicated in the disease and that the effect of the drug is not so strong that it completely stops the process in question (i.e., reducing blood pressure too low is more immediately dangerous than having it too high).

On top of the complexity of the system itself it is wrong to generalize that all cardiovascular drugs treat the same disease, as there are heart attacks, strokes, heart failure, arrhythmia, and others, each with different causes and individual mechanisms to consider. Again, this is contrasted against cancer or viruses, where the cancer or virus itself is targeted directly, while it’s impossible to make a drug to target a stroke since that’s a pathological event, not an agent.

The same hurdles to proving your cardiovascular drug is effective exist for neurological compounds as well, since the nervous system is composed of many different tissues and cell types which are themselves regulated by a wide variety of different neurotransmitters and hormones at different times. And like the cardiovascular system there are many diseases whose pathological mechanisms are very poorly understood and doubly difficult to measure in animals or cell cultures, since you can’t readily ask a mouse if he’s depressed in the first place, let alone figure out if your drug has made him feel better. So again, not only would the FDA have to be convinced that your drug acts on the mechanism you’ve targeted, but you’d have to show them that the mechanism itself causes the disease.

There are probably some cardiovascular or neurological drugs that have a well-described effect on a known target that has been directly implicated in a specific disease, but EVERY cancer and antiviral drug candidate has that advantage already, so it’s not surprising at all from a pharmacological perspective that those drugs are quicker to get approved.

The solution is simple: Merge the categories so as to minimise the variation. For instance, take away the renal bit of "cardiovascular and renal" and add it to the "Oncology" bit creating "Renal and Oncology", repeat until they're all about the same height.

Those stupid doctors in their white coats saving lives of billions of people think they're better than economists.

The FDA is currently holding the most cost effective medical test in history,23andme, for ransom. This is either paternalism or corruption. Privatize most or all of it.

Comments for this post are closed