Is NIH funding seeing diminishing returns?

Scientific output is not a linear function of amounts of federal grant support to individual investigators. As funding per investigator increases beyond a certain point, productivity decreases. This study reports that such diminishing marginal returns also apply for National Institutes of Health (NIH) research project grant funding to institutions. Analyses of data (2006-2015) for a representative cross-section of institutions, whose amounts of funding ranged from $3 million to $440 million per year, revealed robust inverse correlations between funding (per institution, per award, per investigator) and scientific output (publication productivity and citation impact productivity). Interestingly, prestigious institutions had on average 65% higher grant application success rates and 50% larger award sizes, whereas less-prestigious institutions produced 65% more publications and had a 35% higher citation impact per dollar of funding. These findings suggest that implicit biases and social prestige mechanisms (e.g., the Matthew effect) have a powerful impact on where NIH grant dollars go and the net return on taxpayers investments. They support evidence-based changes in funding policy geared towards a more equitable, more diverse and more productive distribution of federal support for scientific research. Success rate/productivity metrics developed for this study provide an impartial, empirically based mechanism to do so.

That is by Wayne P. Wals, via Michelle Dawson.

Comments

Sadly, the larger story is why medical research has largely been so ineffective over the last few decades. The rule is hundreds of millions of dollars for small returns. There has been no real improvement in the treatment of the 'big' conditions: cardiovascular disease, cancer, mental health. Why? At my university, I have seen fellow researchers spend 25 years receiving receiving grants and publishing papers on diabetes or alzheimers...and what do we have to show for it?

While I agree with you in the main, in fact, with the big diseases you've seen incremental advances that have for example in the last generation made cancer 25% or so more survivable (arguably because of earlier detection, which is a bit of a cheat, as Prior Approval I think once said), and heart disease is also more survivable due to angioplasty / stent advances. To paraphrase Jeffrey Sachs in another context, what science needs is a "big push" to break through the Great Stagnation barrier. Mo' & better patents! Future generations will label anti-patent AlexT as a baby killer! The Taberrok curve is junk economics!

Bonus trivia: does bug spray cause Alzheimer's? One junk science article said so, then was contradicted, by the same author, a few years later. Google "Pyrethrum" and see Wikipedia.

In regard to cancer, incremental improvements in treatment have added up over time. The ratio of deaths to new diagnoses was 0.63 in 1970 but about 0.37 in 2010. Also, they have treatments which contain the side-effects of chemotherapy. A dear cousin of mine went through chemotherapy for non-Hodgkins lymphoma in 1977 and 1978. It was a wretched physical trial (nausea, etc) and left his immune system compromised for years afterward and he has some sequelae which put limits on physical activity to this day. Our most recent tour with chemotherapy was in 2007. The anti-nausea drugs and the immune-system boosters are pretty effective. Just some abdominal pain and hair loss.

1 early detection of cancer is not cheating
it is preferable and more difficult than
late detection
2 we are so tired of hrc but it has not affected
standing ability
3 no comas los Doritos anaranjados.
estudia español con Super Holly en el YouTube: ella te animará

If you actually understand what is going on you would understand what was meant by the statement that early detection as a indicator of better cancer survival is a cheat. If you get cancer and are going to die from it in 6 years; under the old system your cancer would be discovered when you had already had it for 4 years and you would die from it two years later. But today thanks to better detection your cancer is discovered in 6 months but you still die from it but 5 1/2 years later. Same life expectancy! BUT the rule of thumb is a 5 year survival is considered "surviving" your cancer. So even though your survival rate didn't change the stats look really good and it "looks" like we are making progress. We need a new or different measure of survival of cancer, something more honest and meaningful.

The other thing to remember that standing still is also a major medical success. Our population has gotten a lot heavier and that drastically reduces lifespan for things like diabetes, cancer, COPD, etc. Our population also has become ever more atomized which results in increased compliance difficulties (e.g. eating during chemo is MUCH easier when you can have somebody else prep food that you like when you want it or getting rides to & from chemo by friends from church, your golf buddies, or the local Kiwanas club).

Likewise, we have massively more dual, triple, or higher survivors these days. I see far more patients who have survived breast cancer, a STEMI, and a stroke than a comparable doc would have seen 25 years ago. Back in the day each of these could be fatal very quickly. Today we can have SERMs for the cancer, a bunch of drugs to prevent pathological remodeling of the heart, and clot busters. Give a patient like this Alzheimer's and they will die much sooner than the traditional patient who managed to live long enough to get Alzheimer's. 25 years ago, Alzheimer's patients had far fewer comorbidities.

And this is a dark actuarial secret. Treating one illness means that the patients live long enough to get a second. If we do better at cancer at keep death off for five more years in a patient, there are good odds that the patient will develop CHF or something else. CHF patients with previous cancer are much harder to treat and die faster than pure CHF patients.

The real way to measure population health is with death rates. Of the top 10 causes of death in America, the only one moving in the wrong direction is suicide. Even for those with only minimal improvement we are looking at lower respiratory disease (which makes sense given what WWII did to smoking rates), accidents, and Alzheimer's.

The evidence suggests that for older people lifespan is higher for the "overweight" than for the "normal"; indeed for the mildly "obese" than for the "normal". You have to be pretty damn fat before your lifespan is as short as for the "underweight'.

Whether such correlations imply cause, Lord knows.

Yes, perhaps skinny people correlate with poor people who can't afford good food and drink a lot of cheap alcohol?

Bonus trivia: the popular "Emperidor Light" brandy in the Philippines is actually cheaper than many brands of bottled water. It's about $1.25 a liter, with about 27.5% alcohol content.

I have seen fellow researchers spend 25 years receiving receiving grants and publishing papers on diabetes or alzheimers...and what do we have to show for it?

Alzheimer's sadly has been very resistant to new discoveries that translate into effective therapies, but that's a bit of an outlier, no?

Would you really want to face diabetes, cancer, heart disease or even Alzheimer's today if your doctors were limited only to knowledge & treatments from 25 years or earlier?

Alzheimer's treatment just had some good news:

https://thedoctorweighsin.com/is-this-new-alzheimers-drug-really-a-breakthrough/

Maybe not the magic bullet as the article notes, but good news for sure.

I don't know. Honestly. For most cancers, the most typical chemo and radiation treatments have been around for decades and death rates haven't budged much (some, like colon cancer, are creeping up). For heart disease, we do a lot of bypasses and implant a lot of stents, but it's not clear those are helping. The same goes for all thosee cholesterol-lowering drugs. Arthroscopic surgery seems a great boon, but....:

https://www.painscience.com/biblio/fascinating-landmark-study-of-placebo-surgery-for-knee-osteoarthritis.html

And so far, don't think new Alzheimer's treatments have moved the needle at all.

I don't know. Honestly. For most cancers, the most typical chemo and radiation treatments have been around for decades and death rates haven't budged much (

https://seer.cancer.gov/report_to_nation/survival.html 5 year survival rates for all cancers have improved. Even if you're only using drugs that have been around for 25+ years I think there's still a material difference. if nothing else doctors will have 25 more years experience with a drug.

Reality is you do know. You would never get back in a time machine and bring your cancer to the world 25+ years ago for treatment. You are correct in that there are always individual bad ideas that come and go so in a very limited sense it might be better to go back in time 25 years and avoid a current bad idea (assuming you can really tell what those bad ideas are). But even here it's a bit silly. You can always refuse a new drug or idea today and insist the doctor treat you with older alternatives. I can't see any support that the last 25 years hasn't made non-trivial progress in a host of serious diseases. Whether it's enough progress given the amount of R&D tossed at it, well that's another issue.

"5 year survival rates for all cancers have improved": but such statements might be worthless. If, for a particular cancer, treatment improves not a whit, but detection occurs on average one year earlier, the five year survival rate will jump smartly upwards. But, in truth, no extension of life has been achieved.

Any intelligently critical thinking would surely lead to a search for measures of success that are a good deal less ambiguous than five year survival rates.

For most cancers, the most typical chemo and radiation treatments have been around for decades and death rates haven't budged much

Your first statement is wrong and the next misleading. You're going to see more cancer diagnoses as fewer people are dying of infectious ailments and heart and circulatory disease. Compare the deaths to new cases. The ratio in 2018 was 0.35. In 1970, it was 0.63. As we speak, the only sites where the ratio exceeds 0.63 in 2018 are as follows: esophagus, pancreas, liver, lung, ovary, and the brain. About 20% of all cancers diagnosed originate at these sites.

1. Grant writing is a skill, large organizations have entire staffs devoted to the task
2. Institutions associated with universities- “publish or perish,”
3. The value of the research, the output, difficult to assess, many million dollar grants may lead nowhere, smaller, highly targeted grants may lead to valuable outcomes

Medical breakthroughs, like most of science, is usually the work of a young man, on his own, ignoring what he is supposed to be doing and flying in the face of what he was taught a few years ago.

The whole grant process means that larger and larger grants go to older and older academics who are established and mainly interested in defending whatever their PhD was about.

Naturally those million dollar grants produce little that is worthwhile. A lot of smaller grants to newly minted PhD students would probably work much better. Except now doctors have to go through endless hoops and Ethical Clearances and committees before they are allowed to do a damn thing. Modern regulations would have killed aspirin and penicillin. They would also have stopped pretty much every major medical breakthrough in the past century. So it is no wonder we are not seeing new breakthroughs. The whole system could have been designed to prevent it.

The problem is, I was told, that you produce excellent work at negligible cost. You must stop: we are judged on our grant income not the quality of our publications.

Pfizer just raised drug prices so something should be coming soon I think.

And you clearly don’t understand what pharma companies do.

Hint: it’s shepherding drugs through phase 1-4 clinical trials, maintaining manufacturing expertise, funding acquisitions of start ups that do the actual research and having an army of lawyers to defend from spurious litigation.

Don’t worry though, the lobbyists, corporate attorneys, politicians and class action lawyers will get rich off their take.

I somehow doubt you’ll soon be posting support of limiting class action suits and deregulating drugs.

Maybe someday we can eliminate all medical advances by refusing to acknowledge intellectual property.

My goodness, defensive are we? Relax. I was making a joke!

As a jnj shareholder I certainly would like to limit punitive damages, if not all class actions.

I’m not sure diminishing returns is the right way to think about it — investigators from elite institutions are receiving grant money based on prestige rather than merit. This misalllcation of resources leads to diminishing returns but that is burying the lede.

About 20 years ago, the Royal Statistical Society's editors published a letter reporting they'd been experimenting with blind reviews (something they hadn't done before) of submitted papers in response to complaints like that. They contended that the result of this policy was that obscure scholars were less likely to have been published. Don't know if they were lying or not.

This article cites research claiming that blind submissions removes some of the prestige effect.

You do get another problem, though, which is that NIH funding decisions are made by experts in narrow sub-areas and people can often recognize who is submitting the grants even without names attached in a way that is not true for something like a piano recital. Right now you are expected to remove yourself from the discussion if a grant involves your co-authors, friends, or enemies. That would be difficult to square with blind reviews — you could probably automate a system to find and notify co-authors but it would be tricky.

You tend economize your work when the next dollar funding it is uncertain. When I was a grad student and post-doc in chemistry, money was always an issue, and result play a big role in getting grants and renewals of grants from the NIH and the NSF. The incentive structure explains the inverse correlation- you work harder to get the money, and less hard when you have plenty of it.

Diseases conquered by science earlier were simpler, simpler to identify and simpler to cure (not simple, just simpler). Cancer, diabetes, heart disease, these are not diseases with a single cause or a single cure, not in one person much less the entire human population. If we all had the same genes, but we don't. If we all had the same diet, and body fat, and climate, and exposure to potential toxins, but we don't. One treatment produces great results in one person, and nothing in another. Indeed, in another, the treatment may make matters worse, much worse. On the other hand, even Bill Gates is warning that the next pandemic is coming. And nothing focuses the mind (and money) quite like mass deaths.

It's not even clear that Type II diabetes is a disease. Cancer is probably a label stuck on many different diseases. With heart disease/CVD the problem may well be that the Cholesterol Hypothesis is bunkum.

Prof. Phillip Johnson of Boalt Hall had an interesting argument many years ago that federal funding of scientific research generated artificial orthodoxies which tended to distort and disfigure the course of scientific discovery. He got interested in the issue when he was retained as an attorney by Dr. Peter Duesberg.

\/- "They support evidence-based changes in {federal} funding policy geared towards a more equitable, more diverse and more productive distribution of federal {taxpayer} support for scientific research... metrics developed for this study provide an impartial, empirically based mechanism to do so."

B.S.

NIH funds are seized from taxpayers and arbitrarily dispensed by inherently political decisions ... not by some mystical objective process, readily at hand from economists. Government does not and cannot operate objectively and efficiently.

Public Choice Theory deeply applies to NIH.

NIH should not exist in a just society. (But NASA scientific productivity has been so great for average Americans -- that we should triple its budget /s)

This discussion is not new.

NIH itself found the same thing a couple of years ago.
http://www.sciencemag.org/news/2016/12/nih-discusses-curbing-lab-size-fund-more-midcareer-scientists
"A new analysis by the National Institutes of Health (NIH) in Bethesda, Maryland, reinforces previous studies showing that productivity may lag as labs get bigger. The results suggest that NIH could fund thousands more researchers without any drop-off in scientific output if it capped the total number of grants an investigator could receive, an NIH official says."

More recently,
https://www.statnews.com/2017/06/26/nih-grants-biomedical-research/
"This spring, the National Institutes of Health announced a dramatic change to the way it would support biomedical research in the United States. NIH Director Francis Collins offered a bold new approach called the Grant Support Index (GSI) that would bolster funding to early-career researchers and limit the number of major NIH research projects that any individual could oversee to three. This approach to funding was proposed in part to grow a larger research community, and in part to ensure that individual researchers were able to responsibly oversee their funded work.

But then Collins talked to advisors at the NIH and changed his mind. Within a month, the GSI was out, and the Next Generation Researchers Initiative — essentially the current flawed system — was in."

What happened?
"After the initial announcement, he turned to his advisory committee, a group of scientists selected largely from the best-funded research universities in the U.S. They heard from members of the research community that the GSI would prevent well-funded researchers from developing new team science projects, and that some researchers would even leave the U.S. to pursue their work elsewhere. They heard that not all well-funded labs were the same, and that it was unclear why the number of major grants per researcher would be three. They argued for more study of the issues, which will take years."

If researchers left the US for better offers elsewhere while the NIH funded different researchers in this country, wouldn't that mean that the NIH was accomplishing:
1) An increase in the total number US-trained researchers receiving funding?
2) An increase in the total number of quality research being conducted?
3) The spread of high-quality research practices to more and more widely dispersed labs/universities/nations?

These should be reasons in favor of the new funding guidelines, not to delay implementing them.

Grants have several competing drivers. First is political considerations. If the today's crisis is climate change everything not nailed down goes that way, also spreading the wealth around congressional districs. Second is CYA. Known, safe researchers at big name institutions are great for CYA. Last, but not least is the scientific prospects. Science tends to go in bursts followed by slower incremental advances. If the NIH guys knew the secret they wouldn't be handing out grants!
Perhaps reducing the near 100% funding of some departments by NIH with more clever private participation would help change focus.

Congress appropriates NIH funding by institute. Just as funding has grown dramatically so too has the number of institutes. https://www.nih.gov/about-nih/what-we-do/nih-almanac/appropriations-section-1 Why would we expect Congress to appropriate funding rationally? The same interest group pressures that apply to every other agency apply here as well. Perhaps more so. NIH has been saddled with huge political statement appropriations such as the $1.8 billion Beau Biden cancer moonshot for which purposes had to be invented and which will never produce anything of value. See: https://www.cancer.gov/research/key-initiatives/moonshot-cancer-initiative/blue-ribbon-panel/blue-ribbon-panel-report-2016.pdf Moreover, over the years NIH grants are often just a fraction of a research programs total funding. Earlier this year, for example, Standford made a much ballyhooed announcement of a successful cancer vaccine for mice. NIH had a grant in the mix but support was also provided by Stanford's Department itself, the Leukemia and Lymphoma Society, the Boaz and Varda Dotan Foundation and the Phil N. Allen Foundation. Two lead researchers also had related commercial interests. Such arrangements have become increasingly prevalent What does NIH return even mean in such a situation? Then we have a certain strain of congresss critter that wants massive funding increases for medical research but lives in mortal fear that it might result in a product that could be licensed and produce profits for someone. One has to wonder how many researchers with promising solutions are avoiding any government funding at all so as to protect themselves from being hauled before a panel of grandstanding gasbags hellbent on making the evening news with anti-business soundbites. And research discoveries and advances advance apace anyway: https://interestingengineering.com/the-16-most-remarkable-healthcare-innovations-events-and-discoveries-of-2018-for-world-health-day

Small institution implies small grant implies small sample size implies high variance. The scientific publishing industry has a filter (p<.05) that skews to publishing high-variance studies. Any measure of "scientific impact" based on publishing should account for this before it is believed.

I question how they evaluate the quality of research. Citation impact has numerous flaws. For one thing, some fields just have more people in them then others--an impact factor that's huge in paleontology would be laughably small in medicine, to give an extreme example. Within medicine some branches are simply more popular than others, and will naturally have a higher impact factor than others.

As for publication number, that's not a valid way to evaluate scientific output. Darwin published comparatively few works, yet he revolutionized several branches of science. Further, a focus on the number of papers published quite obviously incentivizes small-scale research. Who wants to spend 20 years compiling data to provide a real understanding of a field, when those 20 years could be spent getting 20 publications or more?

Until the question of how to evaluate scientific output is fully understood, assessments like this simply aren't valid. At minimum, we must address the fact that there are major issues with our evaluation, rather than simply assuming that these criteria are sufficient.

Comment contains numerous flaws.
--The comparisons are between work in the same fields with the main variables being funding level of the lab and site.
--Picking Darwin as example does not take into account the many thousands of counter-examples.
--Relevance of this 19th century example is far from clear.
--Darwin published extensively:
1829–1832. [Records of captured insects, in] Stephens, J. F., Illustrations of British entomology[1]
1835: Extracts from Letters to Henslow (Read at a meeting of the Cambridge Philosophical Society on 16 November 1835, with comments by John Stevens Henslow and Adam Sedgwick, and printed for private distribution dated 1 December 1835.[2] Selected remarks had been read by Sedgwick to the Geological Society of London on 18 November 1835, and these were summarised in Proceedings of the Geological Society published in 1836.[3] Further extracts were published in the Entomological Magazine and, with a review, in the Magazine of Natural History.[2] A reprint was issued in 1960, again for private distribution.)
1836: A LETTER, Containing Remarks on the Moral State of TAHITI, NEW ZEALAND, &c. – BY CAPT. R. FITZROY AND C. DARWIN, ESQ. OF H.M.S. 'Beagle.'[4]
1838–1843: Zoology of the Voyage of H.M.S. Beagle: published between 1839 and 1843 in five Parts (and nineteen numbers) by various authors, edited and superintended by Charles Darwin, who contributed sections to two of the Parts:
1838: Part 1 No. 1 Fossil Mammalia, by Richard Owen (Preface and Geological introduction by Darwin)
1838: Part 2 No. 1 Mammalia, by George R. Waterhouse (Geographical introduction and A notice of their habits and ranges by Darwin)
1839: Journal and Remarks (The Voyage of the Beagle)
1842: The Structure and Distribution of Coral Reefs
1844: Geological Observations on the Volcanic Islands visited during the voyage of H.M.S. Beagle
1846: Geological Observations on South America
1849: Geology from A Manual of scientific enquiry; prepared for the use of Her Majesty's Navy: and adapted for travellers in general., John F.W. Herschel ed.
1851: A Monograph of the Sub-class Cirripedia, with Figures of all the Species. The Lepadidae; or, Pedunculated Cirripedes.
1851: A Monograph on the Fossil Lepadidae, or, Pedunculated Cirripedes of Great Britain
1854: A Monograph of the Sub-class Cirripedia, with Figures of all the Species. The Balanidae (or Sessile Cirripedes); the Verrucidae, etc.
1854: A Monograph on the Fossil Balanidæ and Verrucidæ of Great Britain
1858: On the Tendency of Species to form Varieties; and on the Perpetuation of Varieties and Species by Natural Means of Selection (Extract from an unpublished Work on Species)
1859: On the Origin of Species by Means of Natural Selection, or the Preservation of Favoured Races in the Struggle for Life
1862: On the various contrivances by which British and foreign orchids are fertilised by insects
1865: The Movements and Habits of Climbing Plants (Linnean Society paper, published in book form in 1875)
1868: The Variation of Animals and Plants under Domestication
1871: The Descent of Man, and Selection in Relation to Sex
1872: The Expression of the Emotions in Man and Animals
1875: Insectivorous Plants
1876: The Effects of Cross and Self Fertilisation in the Vegetable Kingdom
1877: The Different Forms of Flowers on Plants of the Same Species
1879: "Preface and 'a preliminary notice'" in Ernst Krause's Erasmus Darwin
1880: The Power of Movement in Plants
1881: The Formation of Vegetable Mould through the Action of Worms

Underlying tools, methods, and techniques underscoring the post-genome, post-crisper era and more automated sequencing techniques should help returns on research investment. Basically, 1972-2002, we see diminishing returns, based on increasing levels of cutting edge research, predicated on 1970's techniques. Post 2002, and more importantly, post 2016 offers much promise.

There have been breakthroughs in cancer including Car-T cells, which have recently put a woman's stage 4 breast cancer into remission where using this therapy for solid tumors hasn't been very successful.

There are several stem cell therapies for CHF that have shown promise in recent trials and at least two university labs are working on a "heart patch" of stem cells that researchers think will reverse CHF around 2020-2025 where medications would no longer be needed. After decades of research, all of these diseases will be partly or completely cured in the 2020s.

For James' comment, that's like saying the amount of supreme court cases a Justice works on is irrelevant. You measure what you can. No matter how stupid you look, someone in America will pay you for data.

When Emory received its first funding in 1997, the university had $11.3 million in HIV/AIDS research funding and ranked #23 in the nation. Now, Emory ranks sixth in the country for HIV/AIDS NIH funding, with $63.8 million in HIV/AIDS research support.

Yet no cure has come about. Instead there is a focus on lessening costs of HIV healthcare. Meanwhile Brazil steals American patents to save their own people.

"[...] less-prestigious institutions produced 65% more publications and had a 35% higher citation impact per dollar of funding.""

So what? This is perfectly normal. Here "dollars of funding" means "dollars paid by the NIH", as is clear from the context. But researchers have other sources of financing for their research -- their university, their hospital, other public and private agencies... If researchers do not get NIH funding or get less NIH funding, they will still do research using other available funds, and if output was proportional to the fund they use, we should see an higher output by NIH-funded dollar since the total funding has gone down less that the NIH funded. It is just elementary arithmetic, and the article doesn't show any diminishing return of dollars in research.

By this I doesn't want to say that the NIH is doing a good job allocating money or anything like that -- I have no idea on this question besides the standard opinions that anyone may have ... Simply I find this article unconvincing.

My wife, a professor in the Biomedical Engineering Dept. at Duke, is involuntarily screaming at me after I read this post to her. She's saying something like...big science is more of a business than ever, "everything is more expensive." "Techniques are more expensive." We are not just doing "PCR's"; we are doing "fancy multiplex PCR's; they take more time and money." "You want to buy the fancy microscope so you can try it too." Buying equipment is fraught with huge markups. "Why do you think I buy glassware from Ikea? $10 for two or $100 a piece at Fischer Sci.

I was diagnosed with COPD , four years ago now. I moved to USA two years ago in the hope my health would improve
and my doctor has given me Seebri Breezhaler, it's an inhalation powder 44 micrograms. I take it once a day in the morning, but it was unable to help me breath properly. Earlier this year my sister who lives in New Delhi India Purchased COPD herbal remedy from BEST HEALTH HERBAL CENTRE and they told her that their COPD herbal remedy only takes six weeks to reverse COPD completely, which she didn’t’ believe, but she had no choice than to purchase, because she knew breathing was so difficult for me. When I received the herbal remedy, I only used it for six weeks, it was so shocking to see the outcome, I was able to breath normal, no more shortness of breath, no more dry cough, chest pressure stop and my COPD was fully reversed.
I know is hard to believe but am a living testimony, There is no harm in trying natural herbs supplements.

Comments for this post are closed