Do researchers suffer from winner’s curse?

by on October 10, 2008 at 5:07 pm in Science | Permalink

OH NY GOD…READ THIS!!!!!!!!!!!!!!!!!

From The Economist:

In economic theory the winner’s curse refers to the idea that
someone who places the winning bid in an auction may have paid too
much…The same thing may be happening in scientific publishing, according
to a new analysis. With so many scientific papers chasing so few pages
in the most prestigious journals, the winners could be the ones most
likely to oversell themselves–to trumpet dramatic or important results
that later turn out to be false. This would produce a distorted picture
of scientific knowledge, with less dramatic (but more accurate) results
either relegated to obscure journals or left unpublished.

My colleague Omar Al-Ubaydli was part of this work.  Fortunately, it is mentioned on the very cover of the magazine.  Nonetheless I believe it is true and it is most true for "hot" fields.  The original paper is here.

JD October 10, 2008 at 5:18 pm

This is a paradox in teaching medical research writing. We tell students to be accurate and cautious in their interpretation of data. But we then have a culture where those who “over-interpret” get the best press. Given that a few top papers can influence tenure and hiring decisions, I am concerned that incentives may be mis-aligned.

Gabe October 10, 2008 at 5:37 pm

Papers that attack the foundation of the logic(or lack thereof) of the public giving a private monoply to the federal reserve bank also tends not to get hyped in the media too much.

Economist who attack the bailout also tend not to do well in companies that are owned in part by GE, Ruppert Murdoch or Warren Buffet among others.

Andrew October 10, 2008 at 5:47 pm

Markets in everything?
http://www.jnrbm.com/

I propose the B.S. prize. Get a million bucks for calling B.S. on someone published in Science or Nature. $500k for JAMA and $2mil for a Nobel winner.

kurt October 10, 2008 at 6:01 pm

Can researchers on winner’s curse suffer from winner’s curse as well?

Dave October 10, 2008 at 6:43 pm
Brainwarped October 10, 2008 at 7:05 pm

I thought the winner’s curse says a person who wins an auction paid the most, therefore, they feel like they lost because they paid more than the next highest bidder, IE they got cheated out of a small amount of money.

Then logically, the winner’s curse applied to papers would be that published papers present more information than was necessary to get published, IE they could have published 2 papers instead of the one.

Therefore, I refuse to read the paper because I disagree with the conclusion from the snippet Tyler posted… Why does the extra information have to be incorrect? Doesn’t the Journal check the validity of the information before it is published? (And I’m not talking about the economist.)

Andromeda October 10, 2008 at 7:32 pm
Don the libertarian Democrat October 10, 2008 at 10:14 pm

Who Says Capitalism Is Dying?
Some good news for the weekend:

http://www.nytimes.com/2008/10/11/world/asia/11china.html?hp

“BEIJING — Chinese leaders are expected to allow peasants to buy or sell land-use rights for the first time, a step that could draw hundreds of millions of farmers more firmly into the city-centered market economy.
The new policy, which is being discussed this weekend by Communist Party leaders and could be announced within days, would be the biggest economic reform in many years and would mark another significant departure from the system of collective ownership and state control that China built after the 1949 revolution.”

Read on.

Who says capitalism is dying?

AZ October 10, 2008 at 10:32 pm

@Andrew at Oct 10, 2008 5:47:34 PM

>I propose the B.S. prize. Get a million bucks for calling B.S. on someone published in Science or Nature. $500k for JAMA and $2mil for a Nobel winner.

We now know who controls the $700 billion and what he plans to do with it … :)

Thorfinn October 11, 2008 at 1:42 am

One way to get around this is to be a Bayesian and adopt informed priors on issues. That way, instead of looking for a “p value of such and such” you start demanding extraordinary evidence for extraordinary claims.

Andrew October 11, 2008 at 5:03 am

“We now know who controls the $700 billion and what he plans to do with it … :)”

Don’t I wish. It would be used for just about the opposite of its current destiny. On that note, it seems like there ought to be some kind of do-over on the election now that the government is going to have vastly different powers. I mean, why elect a socialist or a warrior when you’ve already enacted socialism and there’s no way in hell you’ll have any money to fight wars.

Brain,

“Then logically, the winner’s curse applied to papers would be that published papers present more information than was necessary to get published,”

Yes, and then the point is that the extra information is NOT TRUE, or at least sketchy, and if everyone agreed to not stretch, then the playing field would be even.

It’s kind of like doping in sports where doping really helps.

So, I suppose the direct analogy to winner’s curse as explained would be that auction winner’s beat the next highest bidder by an amount paid in counterfeit money.

Barkley Rosser October 11, 2008 at 7:22 am

To clarify, in this case it is not the “winners” in the sense of the researchers who are cursed or suffer. It is knowledge or academia or the journals that do from “buying” faulty research.

I think it was Emily Ostrom, not Oster, whom Tyler praised, and Ostrom is certainly admirable, not a case of a winner’s curse in regards to her research.

Econ Journal Watch may be the nascent economics journal of negative results.

AZ October 11, 2008 at 8:04 am

Barkley – there are 7 past MR posts about Emily Oster, and zero about Elinor Ostrom (she has been mentioned in the comments). I tried to copy and paste the 7 posts that mention Oster but the TypePad spam filter thinks it’s spam.

Xabier October 11, 2008 at 9:15 am

I propose an alternative (and more accurate) title for the article on the winner’s curse. The title should be: “Why this Article is a Good Example of Why Current Publication Practices May Distort Sicence”.

happyjuggler0 October 11, 2008 at 12:24 pm

I’m trying to figure out who the NY GOD!!!!! is. Could you mean Brett Favre? Dave Letterman? Martin Scorcese? Robert De Niro?

But why would they want to read it?

With that thought in mind, perhaps you are referring to Taleb? Or Easterly? Or Mundell?

Oh wait, you are imitating Higgins from Magnum PI!!! How silly of me.

Anonymous October 12, 2008 at 8:23 pm

Brainwashed and Andrew are confusing Winner’s Curse with Buyer’s Remorse.

Buyer’s Remorse is indeed the phenomenon where a buyer realises ex post they could have got the goods for only a slight margin more than the second bidder – the seller has extracted a rent. But it doesn’t mean they can’t make a profit from the purchase.

Winner’s Curse arises where the value of the good is uncertain and different bidders have different ex ante estimates of its value. The winner will be the most optimistic valuer, who by definition will be the one most likely to overestimate the true value. The publishing bias discussed here is that of Winner’s Curse, not Buyer’s Remorse.

Frank Dean October 15, 2008 at 12:38 am

There’s a nice paper on this topic from 1992 by DeLong and Lang.

http://www.j-bradford-delong.net/movable_type/archives/000620.html

This also provides a nice example of another huge problem in academia: nobody has any idea what’s going on outside each person’s specialty, so there’s massive duplication of effort.

hellen May 14, 2009 at 9:00 pm

all the things have two sides

jack May 14, 2009 at 9:01 pm

Is it realistic?

wholesale stockings November 17, 2010 at 3:43 am

More are waiting a very nice article thank you for the love

ガバペン March 9, 2011 at 3:56 am

Great tips, I would like to join your blog anyway,

Comments on this entry are closed.

Previous post:

Next post: