Why is there so little analysis of NSF economics grant-making?

by on August 5, 2016 at 2:10 am in Economics, Science, Uncategorized | Permalink

As Alex and I were writing our Journal of Economic Perspectives piece on the economics activities of the National Science Foundation, I found it striking how little serious discussion there has been of the topic.  You’ll find only a few scant references in the paper, and some of those are quite old now.  This is part of a more general reluctance of economists to use economic tools to understand themselves.

I worry about the discourse-chilling effects of having government fund social science in this way, just as I worry about the implicit censorship effects on those who might seek to someday chair the Council of Economic Advisers.

On the NSF, let me be clear.  I had a long meeting with some of the decision-makers at the economics division, and I found them to be extremely helpful and open and fair.  I have little doubt that if I, now on record as somewhat of a critic of current procedures, submitted a grant request, it would be evaluated with full objectivity.

The problem is, the rest of the world doesn’t know that.

Addendum: You will note this section of the paper:

Another public good the NSF might fund is simply a study of which of its previous expenditures on economics have had the greatest marginal value-added. Based on conversations with NSF staff in economics, we are not able to identify such a study. One proposal would be to fund such a study and then follow many or all of its recommendations; after all, the NSF presumably believes it is capable of generating useful research results with practical implications.

Here John Cochrane comments on our paper.  If you are wondering, this was my favorite part of the paper:

A related question is to consider the elasticity of supply for quality economic research, with respect to wages or payment. Over the last few decades, many in the economics profession have concluded that tax cuts for high earners, at current tax rates, have relatively small effects on labor supply; for example, that argument is often cited as one reason why the Bush tax cuts passed into law in 2001 yielded disappointing economic results. Yet when it comes to NSF grants, there is often the implicit presumption that the elasticity of supply for economic research with respect to additional government grants is relatively large. We are agnostic on the elasticity question, but without understanding this issue, it is difficult to evaluate NSF bang for the buck, and thus economists should not be so confident about the efficacy of this funding.

File under: “Elasticity Optimism for Me, but not for Thee.”

1 BC August 5, 2016 at 2:37 am

“Another public good the NSF might fund is simply a study of which of its previous expenditures on economics have had the greatest marginal value-added.”

1) Are the economics questions about NSF funding of economics research different from those about NSF funding of research more generally, e.g., research in sciences and engineering? This marginal value-added question would seem to apply equally to scientific and engineering research. Maybe, economists would be less reluctant to use economic tools to study non-economics research funding after which those same tools and methods could be applied directly to studying economics research funding.

2) One aspect of research funding that doesn’t seem to be mentioned publicly very often is that research funding is as much, if not more, about subsidizing graduate studies as it is about producing research. At least in engineering and sciences, research grants support many graduate students with research assistantships. My former advisor once claimed that one of the criteria used in evaluating research proposals was the number of graduate students that would be supported. Thus, in trying to understand the marginal value-add, one should include the public good aspect, to the extent that there is any, of graduate education subsidies. Do economics research grants typically include support for graduate students?

2 Edward Burke August 5, 2016 at 2:55 am

Mutatis mutandis:

1) the NEA and Federal arts funding (do performance arts get the bulk of the funding as seems to be the case? [how substantive, that is, is funding not for writing grants but for publishing outlays?])

2) the CPB and Federal funding of public broadcasting (the CPB STILL gets hundreds of millions annually, no matter what NPR stations say at pledge time to conceal the fact: and the entire rationale for LBJ’s “gift to domestic broadcasting” has long since died the death, since the advent of cable and even more so since the advent of the internet). Federal subsidization of public broadcasting has helped undermine news dissemination and narrowed the range of published perspective, someone could argue.

3 prior_test2 August 5, 2016 at 3:43 am

‘I found it striking how little serious discussion there has been of the topic’

Because no one seriously considers economics a science?

4 agm August 5, 2016 at 6:59 am

“Yet when it comes to NSF grants, there is often the implicit presumption that the elasticity of supply for economic research with respect to additional government grants is relatively large.”

If you want to study it to push the boundaries of applied human knowledge, knock yourself out. However, at the level of funding decisions, there’s no real need to for these studies of this sort — the tenure chase guarantees that the demand for funding might as well be infinite.

5 Euripides August 5, 2016 at 7:37 am

Let me spell out what NSF money is perceived to be in the ranks of non-top schools:
It basically provides summer income for researchers from the very top schools for research that they would conduct even if they were not funded.
Who evaluates the projects? Their buddies in the same field but at different top schools.
It’s a nice racket if you can get a piece of it!

6 Eustis August 5, 2016 at 8:05 am

yeah, government bureaucrats casually give away billions of tax dollars every year for “research” of officially unknown (but very likely negligible) practical value to the public good.

Economists viewing this obviously negative “public-good” charade are mostly indifferent, but here we have a few who are somehow totally bewildered by NSF’s actual functioning/purpose and unable to draw any firm conclusions.

“Why is there so little analysis of NSF economics grant-making?” — because the nominal “analyzers” are insiders and ultimately part of the big con game.

7 Willitts August 5, 2016 at 8:52 am

Don’t look a gift horse in the mouth.

Don’t kill the goose that lays the golden eggs.

Or, the overshadowed and ignored portion of Ike’s farewell address:

…there is a recurring temptation to feel that some spectacular and costly action could become the miraculous solution to all current difficulties. A huge increase in the newer elements of our defenses; development of unrealistic programs to cure every ill in agriculture; a dramatic expansion in basic and applied research – these and many other possibilities, each possibly promising in itself, may be suggested as the only way to the road we wish to travel…

But each proposal must be weighed in light of a broader consideration; the need to maintain balance in and among national programs – balance between the private and the public economy, balance between the cost and hoped for advantages – balance between the clearly necessary and the comfortably desirable; balance between our essential requirements as a nation and the duties imposed by the nation upon the individual; balance between the actions of the moment and the national welfare of the future. Good judgment seeks balance and progress; lack of it eventually finds imbalance and frustration.

In this revolution, research has become central, it also becomes more formalized, complex, and costly. A steadily increasing share is conducted for, by, or at the direction of, the Federal government.

Today, the solitary inventor, tinkering in his shop, has been overshadowed by task forces of scientists in laboratories and testing fields. In the same fashion, the free university, historically the fountainhead of free ideas and scientific discovery, has experienced a revolution in the conduct of research. Partly because of the huge costs involved, a government contract becomes virtually a substitute for intellectual curiosity. For every old blackboard there are now hundreds of new electronic computers.

The prospect of domination of the nation’s scholars by Federal employment, project allocations, and the power of money is ever present – and is gravely to be regarded…

Yet, in holding scientific research and discovery in respect, as we should, we must also be alert to the equal and opposite danger that public policy could itself become the captive of a scientific-technological elite.

8 Johnny A August 5, 2016 at 10:10 am

Wow, what an interesting quote. (Seriously; not snark.) Thanks for posting.

9 Willitts August 5, 2016 at 1:46 pm

I recall my amazement the first time I read it. While almost everyone is familiar with the phrase Military Industrial Complex, it appears that most people who use that phrase haven’t read the entirety of his speech in its proper context.


10 Linguist August 5, 2016 at 8:04 am

As a linguist, I would like to add that similar issues can and should be raised about all fields: an economic approach to funding mechanisms could be of great interest to science in general, not just economics.

I have had experiece with the NSF, the European Science Foundation and the ERC. I am struck by
(i) the huge differences among these three systems;
(ii) the complete absence (to my knowledge) of serious evaluations of the costs and benefits of each system.

Examples from my limited experiences as a linguist [those pertaining to the NSF should be taken with a grain of salt, as they might be outdated]:

(i) NSF Linguistics has had a long-standing reputation for:
a. not funding theoretical work; and
b. having what amounts to a queue-based system: submit a project, get rejected, dot all the i’s, resubmit, get funded.

a. has probably been false for many years but the perception hasn’t really changed (at least the last time I checked, a couple of years ago) – hence theoreticians don’t apply much, I think. If true, effects on the field should be discussed (e.g. is there enough of an incentive to engage in ambitious theoretical work?)
b. was true at some point, and it might or might not still be true, but it’s fairly inefficient.

(ii) The now defunct ‘Euryi’ system of the European Science Foundation and the ERC system that replaced it are strikingly different from NSF grants in that
a. in poor fields like linguistics, they give way more money;
b. they involve broader (less specialized) panels, which might facilitate cross-disciplinary work;
c. they have a clear policy of encouraging risk-taking – which seems to me NOT to be the case at NSF, which in my partial (and possibly outdated) experience is a bit conservative [this might be unfair and should be checked, however].

(iii) More strikingly, Euryi had an award-based system: applicants submitted a project, but if funded, they could use the money with huge flexibility, under the control of their home institution.

ERC does the *opposite*: the money is supposed to be used *exactly* as outlined in the project, unless explicit changes are made. Organizational and financial costs seem to me to be enormous: expenses are typically checked at 4 different levels, and the ERC, national institutions and group leaders must employ people who are specifically in charge of cost justification. (In my experience, the people involved are highly competent, but that’s another matter: the *procedures* seem to involve a *huge* waste of resources – one would like to know how much time and money goes into cost justification, for instance.)

This move towards greater bureaucratization is all the more striking since:
a. the ERC system essentially replaced the Euryi system, and could have built on its non-bureaucratic nature;
b. after something like 100 Euryi-style grants, I don’t know of any problems that were raised due to insufficient oversight (and I’ve asked);
c. the only convincing response I got from the ERC about its move towards super detailed cost justification is political: scientists in charge are convinced that politicians will never accept an award-based system on a grand scale.
They might well be right, but this only highlights the need for an evidence-based approach to funding mechanisms.

One final point: I once asked some prominent colleagues in economics whether they knew of economic research on funding mechanisms. I got the impression that this was not a hot topic in the field – which is greatly regrettable in view of its importance.

11 jeturcotte August 5, 2016 at 8:25 am

I’m come late to this and have not read the paper to which you are reacting; but I can add an anecdotal thing about NSF grant funding analysis, having personally been involved in a community-based experiment with their (and the patent office’s) public data… something that came out of a project led by an attendee of the Code for NoVA hackerspace (read: community activism through data, computation, design, et cetera, for the improvement of said community often but not exclusively in response to needs local government has or is struggling to meet) and worked on my a slew of students (good for their eventual resumes.)

Point being, we took data from both the NSF and the USPTO. The NSF gives grant money and, in return, requires certain forms be filled out when and if that grant results in the creation of a patent (which often results in the creation of a new business; something many would consider a boon for the economy.) Compliance with this had been routinely dismal. I forget the spread, but we focused on 2012, which included both grants and patents over a slew of years (as often the result of a grant bore fruit many years later.)

In the end, through some machine learning, some regex, and some human eyeballing to vet the results, we quite literally found a roughly 800% increase in the number of patents that were the result of NSF grants. In the forms submitted, there were a slew of mis-formatted IDs for the grants given… misplaced ones (in the wrong fields), et cetera. Being able to correctly link these erroneously formatted submissions with the original grants was quite fantastic, and spurred some alterations in the forms within the NSF to lessen the probability that patent winners would misreport the success of the grants they’d received.

Altogether, what I’m suggesting is that there has been SOME (albeit unofficial) analysis, and that it told a better story than some might have feared.

12 anon August 5, 2016 at 8:51 am

Is it because social science, as a percentage of NSF budget, is so small that nobody really cares?


13 Willitts August 5, 2016 at 8:56 am

As a first pass, look to see if the Office of the Inspector General for the NSF has done such studies. If they have not, then suggest such studies or contact the appropriate Congressional committee to call for such a study.

Alternatively, request the data through the Freedom of Information Act and do the research yourself.

14 rayward August 5, 2016 at 9:01 am

Although Cowen and Tabarrok argue (or do they assume) that NSF funding of economic research isn’t objective, isn’t that the point of NSF funding: that the NSF has no dog in the fight and, hence, won’t pick and choose based on the likely outcome of the research? I’m skeptical of research funded by those with vested interests in the outcome, whether the interests are economic (as when a company funds research the outcome of which will promote the company’s economic interests) or ideological (as when a person funds research the outcome of which will promote the person’s ideology). Cowen and Tabarrok argue (or do they assume) that those with economic interests in the outcome of the research will be objective in funding research because they have an economic interest in the outcome. Of course, that assumes that company’s which fund economic research (or any research) are passive participants, blindly following the research wherever it may lead, even if it may lead the company to abandon a product or strategy in which the company has already made a large investment. Do Cowen and Tabarrok actually believe that? Do they believe a company would blindly fund research that might conclude that the company’s product is causing irreparable damage to the planet because companyies are motivated solely by profits and, hence, a company would abandon a product that would damage the planet since a product that would damage the planet would adversely affect the company’s profits? I’m not arguing that the NSF is free of bias, but compared to the alternatives it is.

15 Willitts August 5, 2016 at 10:08 am

The issue isn’t (or shouldn’t be) whether research is expected to reach a certain conclusion. Rather, that the value of the conclusion, whatever it is, is worth its cost. An important part of the grant request is to explain how and why the research will advance the science or fill crucial gaps. It’s also fitting to demonstrate that there is a market failure in the provision of such valuable research such that the private sector will not undertake it.

16 Albigensian August 5, 2016 at 10:11 am

Has anyone even looked into the diminishing returns of research funding? For surely at some point there are few talented researchers left whose research work is blocked due to lack of funds, and few promising avenues of research that are being neglected due to lack of resources?

17 Edgar August 5, 2016 at 10:47 am

Great articles, both C&T and Cochrane. Cochrane especially rings true with respect to federally funded research generally. There have been few if any looks at the opportunity costs of federal research funding. The last Bush administration’s efforts to undertake meaningful program evaluation failed miserably and the federal research portfolio is now treated as some kind of Lake Wobegon where all grants are above average. The news media today are all clamoring for Congress to fund “Zika response” yet none of the articles even begin to attempt to explain what the money would actually be spent on. The US spent billions on Ebola and yet there is no evidence that any of it did any good and that the outbreak would not have gone away with just the local government efforts already in place. However bad economics research funding might be, “crisis” driven federal funding is likely much worse.

18 Bill August 5, 2016 at 10:48 am

Can someone help me draft a grant request to the Mercatus Center to fund my research of the economic grant making of the NSF?

Shucks, it’s already been done.

I mean, aren’t the posters here beneficiaries of an economic grant making institution for economic research.

Has anyone done a study of the Mercatus Center and it’s research program.

Or, is that not politically correct?

19 JK August 5, 2016 at 10:57 am

I sit on one of the econ-related federal scientific review committees for grants. We are explicitly prohibited from asking questions like, would this study occur without the requested funding? And, what is the value of this project relative to its cost?

20 Willitts August 5, 2016 at 1:43 pm

Thanks for the inside info although I’m a bit astonished that cost benefit analysis isn’t being done, at least implicitly. Doesn’t the importance of the research goal play some part?

21 JK August 5, 2016 at 2:31 pm

Importance, yes, heavily. At least as defined by some combination of the agency and the committee. But not value per se, nor any thinking on the margin as Alex and Tyler are emphasizing here.

22 Bill August 5, 2016 at 10:59 am

Let’s look a little bit at how subsidies in economic research are granted in the real world, and not just competitive grant making world NSF.

So, here’s what I did. I looked at the websites of several conservative economists to find how their research is subsidized.

Direct grants from conservative institutions. Papers published in the magazines of those institutions (this is an in kind subsidy–papers that would not be published in AER get published in a foundation sponsored magazine). Foundation sponsored seminars and fora, with speaking fees. Direct sponsored research by trade associations and firms. Consulting followed by publication.

Look at all the ways that research is funded, and it seems to me the NSF is one of the few places where there is grant applications without regard to political economics.

23 Art Deco August 5, 2016 at 11:15 am

Because it would be awkward if the hogs ended up making the case that they shouldn’t be fed their afternoon swill.

24 wd40 August 5, 2016 at 11:17 am

This is in reply to Cochrane’s comment. Doesn’t a significant proportion of the funds (I do not know the percentage which would be interesting to know) go for the payment of research assistants and other costs associated with research) that one would not expect to be self-funded by academics? The whole idea for government funding of research is that it is an alternative to patents (as pointed out by Alex patents have many problems). Of course,economic research is generally not patentable but it presumably does have valuable spillovers. Summer salary on grants might be a way of discouraging economists from undertaking private consulting with fewer intellectual spillovers but greater income for the researcher. I am not saying that these counter-arguments are definitive, but it does seem that economists should at least consider them before jumping in.

25 Li Zhi August 5, 2016 at 11:42 am

TC believes that the NSF process is blind to any criticism when it comes to awarding grants? That would be astounding if it were true. Homo economicus is alive and well in Virginia.

Comments on this entry are closed.

Previous post:

Next post: