Science sentences to ponder: citations vs. innovations

We demonstrate empirically that measures of novelty are correlated with but distinct from measures of scientific impact, which suggests that if also novelty metrics were utilized in scientist evaluation, scientists might pursue more innovative, riskier, projects.

That is from Jay Bhattacharya and Mikko Packalen in a new NBER working paper and scientific innovation and stagnation.

They point out that Eugene Garfield, the scientist behind the development of citation count, did not think it should be used to evaluate individual scientists.  Overall, citations encourage too much work in crowded, “approaching peak” areas, rather than developing new ideas.  In lieu of citations, the authors suggest using textual analysis to determine how much a paper is building on new ideas rather than on already intensively explored ideas.



Any system based on textual analysis will be quickly gamed,

Yes. As will anything.

I think the core problem is trying to objectively measure something as subjective as the value of someone's contribution to scientific knowledge. This problem is compounded by basing people's careers on the measurement.

Rather than coming up with better metrics, reducing the reliance on any such metrics in hiring, promotion, funding, institutional ranking, etc. might be a better idea. Subjective evaluation of work by a panel of experts, as flawed as that can be, is probably the best mode of making these decisions.

If the panel of experts detects a politically incorrect view, you will run towards the nearest textual analysis machine.

The current system does not appear to mitigate this problem at all.

Textual analysis is precisely the postmodern boost science needed.

Pretty sure this is what you have in mind (from a close reading of your post):

I bet "PATENTS" are not mentioned even once. Sad. '
To draw the analogy with patents, the pioneer base patent invention, while important, and often cited, pails in comparison to the more commercially important secondary patent invention, which is what the marketplace wants and what sells, and what lots of R&D is spent on. Think chunky brick cell phone vs today's slick "Gorilla Glass" touchscreen smartphone. Base patent (parent) vs secondary patents (children, lots more children than parent). And, btw, often the secondary patent holders don't even bother getting licenses from the base patent holder (that's how patent litigation commences, but litigation is expensive). Analogy: in science I bet a lot of secondary scientific papers don't even acknowledge the base stem paper except in passing, and fob off something the base patent implicitly already discovered as their own 'new' and 'radical' discovery. Publish or perish perverse incentives are behind this. Perverse incentives that are common in patents too, since often the "me too" slicker version of an invention gets 100% market share (one reason I favor a longer term of protection for a true base patent pioneering invention). Anti-patent types would view such free-riding as 'competitive behavior' however, one reason we have a Great Stagnation. It doesn't pay to be a real innovator (but it pays to be second).

Realistically people aren't going to waste too much time going through a lengthy and expensive court battle when they can just go to market. It is more profitable for Corning to win over Apple as a Gorilla Glass customer than to go after copycats. Better to chase revenue than to run up legal fees. The main value of a patent as it is today is arguably defensive. If somebody tries to sue you, you look in your patent portfolio and see if you can find leverage to bring them to the table. Unfortunately only big corporations and patent troll law firms can do that. Everybody else pays the cost of extortion or goes under. If the best use of patents are defensive, then the value of patents are trending down and no that has nothing to do with the Great Stagnation. Tesla is a great innovator but their approach to patents is to literally play defense (sue only if the other guy sues) and to open them up to everybody (just don't pretend you are Tesla).

Elon and team realize that patents put up speed bumps on the path to innovation and better to pay up for good engineers than good lawyers to accomplish that:

"Technology leadership is not defined by patents, which history has repeatedly shown to be small protection indeed against a determined competitor, but rather by the ability of a company to attract and motivate the world’s most talented engineers."

@robert, thanks for the substantive response. Nothing you say contradicts my counterfactual with deals with what should be done not with what is being done; all that you say is well known in the IP industry (patents are used defensively). I would also argue Tesla is like the early Google, if you remember, they were patent light (they ended up playing catch up when the bought Motorola's patents for a premium, along with others). Same actually with many companies, including Dell and arguably even Microsoft. Essentially Tesla is a marketing company, not unlike Google, that just happens to build conventional albeit cool electric cars.

A patent system is better than a free-for-all, but I don't think there are many people who would say that the current American patent system is optimal, especially for growth. It favors entrenched players and discourages startups.

The system needs to be adjusted in favor of individual initiative and away from random legal threat.

"A patent system is better than a free-for-all"

Better, one can't help but wonder, for whom? Open source software runs most of the internet's critical infrastructure and has brought affordable internet access to the next billion (and quickly working on the billion after that) via primarily Android. It hasn't slowed the pace of innovation, but has accelerated it. Android manufacturers don't make applesque profits, but I view apple's customers as suckers rather than Android users as freeriders.

(My radical opinion is that an innovator should be able to go out to the garage and start innovating, without a lawyer. And that independent invention is not intellectual property "theft.")

@anonymous - right on bro! I agree 100%. Recall in response to North49, who is talking about software (and BTW I'm against a long term of protection for software patents) that S/W engineer that TC linked to that castigated the open source movement as leading to largely abandoned junk software. Proprietary (trade secret) software is good software.

As to your point, that garage guys should be exempt from being sued, it's a good one, and was fought roughly 25 years ago by small inventors who petitioned Congress and the US PTO to reform the system. They lost ("first to file" rather than the stealthy "first to invent" won in 1995), as today's "first to file" system de facto rewards companies with large in-house patent departments that can file on every conceivable design variation and then some. It's also the "world standard" (globalization theme). One reason in response to this is why "patent pools" have arose, as it's impossible to ascertain who invented what, so you simply pool together the essential and non-essential patents among the various players (a legal form of crude market sharing, at least among IP, one of the few times US antitrust law generally does not apply to market sharing among competitors). For this reason BTW I think QCOM (Qualcomm) is a long-term buy, I like their 5G and CDMA patent portfolios (and as I predicted they won their litigation, based on my reading of their public documents). Bye; thanks for the replies to all in this thread, I think in all the years I've been posting this is the closest we've gotten to a substantive review of patents. I've given upon on TC and AlexT, as they are typical perfect competition advocates which by definition excludes patents and solving for the equilibrium means we all should work at Rev. Malthius' subsistence wages (smile).

@ray this is econ 101 stuff, it's not hard.
What is a private good? Say it with the class:
Good, and?
Which of these apply to ideas?
Okay then.

Anyway, weren't you busy having another trust fund baby these days and didn't have time to 'educate' us anymore. That sounded good to me.

@North49 - "If non-excludable goods are produced by the free market, there will not be enough people paying for the product to warrant firms producing the allocatively efficient quantity. As a result, the market will tend to under-produce. This is a market failure and creates dead weight loss. Often non-excludable goods will be provided by the government." - I'm glad we're in violent agreement about the necessity for better and stronger patents my fiend.

😒 that's an argument for public research, not for fiating a private property interest in a non private good. We can also legislate 'up is down', it doesn't make it so.

Until researchers learn to game the textual analysis.

Good thing researchers are instead using citations because that can never be gamed ever.

Mike wants to "lower America's economy"? Thiago, you'd make a pretty lousy campaign manager. Mueller would've put you in prison years ago.

He probably means power.

And the use of excessive incestuous mutual admiration club rules of “You citate me and I citate you” should also be punished.
Build an index of mutual cross references and much will improve.

These were anticipated long ago--search for "invisible colleges"

As I've stated many times, effective innovation occurs at the margin, which was the once-dominant view. China seems to adhere to that view, taking innovations of others and improving them at the margin. In the U.S., that view has become heterodox, the orthodox view favoring blockbuster innovations. Thus, old innovations are mostly ignored in favor of the blockbuster innovation, which mostly means an innovation in tech that can be quickly monetized. Is that why we can't build an efficient transportation system? Is that why GE, once a leader in innovation, is treading water? Is that why Boeing, once the leading innovator in aeronautics, would develop and build what is a defectively designed aircraft because it was the cheapest and most profitable design? This blog may be called Marginal Revolution, but sometimes I have the feeling that it's a misnomer.

"Overall, citations encourage too much work in crowded, “approaching peak” areas, rather than developing new ideas"

This sounds like what I've described as "fully subscribed" research areas.

It has to be clear and hard….But the most important thing of all…is not to kill the story by working on it. Or else all our labor has been in vain. It’s like walking a tight-rope. Ralph, there it is….We ought all to take an oath not mess up our job.

Textural analysis requires that the people or AI actually understand what they are reading. I challenge the authors of this paper to pick a random issue of Science magazine (AAAS journal) and read all the papers in the Reports section (real not lay journal articles) in the back. They can read and look up all the words, but how many articles do they fully understand. If it is more than a few I would be astounded.

Then understanding at a level where you can determine which are truly innovative is even more complex, because most real innovations require knowledge in other areas to understand the significance.

Science and Innovation have changed in ways that economists don't seem to really understand. Even people who are considered top in their fields can't read and fully understand papers in other real science fields making it difficult to know whether an idea, concept or innovation is significant.

Only in the so-called "social sciences" and humanities is it possible to read and understand the papers by just translating the buzz words into English and that is because there is so little real progress in these fields. The fancy mathematics in some of these areas is more like Astrology which uses great analytical methods on the motions of the heavenly bodies to obtain great precision with no real understanding or knowledge.

Comments for this post are closed