Do Experts Listen to Other Experts?

Not so much:

Organizations in science and elsewhere often rely on committees of experts to make important decisions, such as evaluating early-stage projects and ideas. However, very little is known about how experts influence each others’ opinions, and how that influence affects final evaluations. Here, we use a field experiment in scientific peer review to examine experts’ susceptibility to the opinions of others. We recruited 277 faculty members at seven US medical schools to evaluate 47 early stage research proposals in biomedicine. In our experiment, evaluators: (1) completed independent reviews of research ideas, (2) received (artificial) scores attributed to anonymous “other reviewers” from the same or a different discipline, and (3) decided whether to update their initial scores. Evaluators did not meet in person and were not otherwise aware of each other. We find that, even in a completely anonymous setting and controlling for a range of career factors, women updated their scores 13% more often than men, while very highly cited “superstar” reviewers updated 24% less often than others. Women in male-dominated subfields were particularly likely to update, updating 8% more for every 10% decrease in subfield representation. Very low scores were particularly “sticky” and seldom updated upward, suggesting a possible source of conservatism in evaluation. These systematic differences in how world-class experts respond to external opinions can lead to substantial gender and status disparities in whose opinion ultimately matters in collective expert judgment.

That is from a new working paper by Misha Teplitskiy, Hardeep Ranu, Gary Gray, Michael Menietti, Eva Guinan and Karim R. Lakhani.  For the pointer I thank the excellent Kevin Lewis.

Comments

"These systematic differences in how world-class experts respond to external opinions can lead to substantial gender and status disparities in whose opinion ultimately matters in collective expert judgment." Seems unsubstantiated by the actual study. And what's with the fuzzy language... "can lead"?

Alternative title: you can't bullshit a bullshitter

Respond

Add Comment

Respond

Add Comment

Cognition: an intellective display of volition.

Respond

Add Comment

Did I understand correctly that, when they say that the scores were "artificial", it means that they were made up? If so, I would think that susceptibility to change scores based on the review would be a bad thing, as it means that the scientist would have changed his or her score simply to conform. Am I mistaken?

The whole point of the study is to see whether the reviewers would change their scores when given the scores other "experts" assigned the research proposals. The artificial scores shown to the reviewers were also accompanied by descriptions of the artificial reviewers (whether they are in the same field, etc.). In either case it wouldn't really matter if the scores are artificial (made-up) or real, because any change in the reviewers original score gives an indication of how that reviewer weighs the opinions of other experts (note: the experts could change the score in the opposite direction of the artificial score). It's just that artificial scores are cheaper to generate and allow the authors to manipulate the treatment to which the reviewers are exposed. Whether or not updating based on information from other experts is a bad thing is not something you can tease out from this study.

Respond

Add Comment

"the scientist would have changed his or her score simply to conform."

I don't think that's necessarily the case. Putting a numerical score on a research proposal isn't cut and dried. There are always competing priorities and it's not obvious which should be weighed the most heavily. It would be easy for a reviewer to give a score that they feel unsure about. Then, when they see I different score provided by someone else, they reconsider their original number not because they're just following the heard, but because they were unsure about it in the first place and when they see a number that conforms to the alternative way of rating the proposal, they imagine in their minds that the other reviewer was facing the same problem.

It's a revision initiated by the other expert's opinion, but it's a case of taking that opinion into account - which is entirely appropriate - rather than simply following the heard.

Respond

Add Comment

Respond

Add Comment

If a field is not replicable easily, then the people seen as experts will be skilled in rhetoric rather than actual ability. "Experts" will know this about themselves and other experts in their field, and will therefore feel free to disagree with the other experts in a way that a layman wouldn't.

That may be but has no bearing on this study, since the reviewers were only given numerical scores as added information.

Not sure I understand your point. People were given the chance to update their scores based on input from others, the experts know that they don’t really know anything so they assume same for others so they update less than neophytes who believe the field has true knowledge.

"people seen as experts will be skilled in rhetoric rather than actual ability ". With just a numerical score there is no rhetoric.

"the experts know that they don’t really know anything so they assume same for others"?

Yes, they don’t need rhetoric to know that nobody knows anything.

Respond

Add Comment

Respond

Add Comment

Respond

Add Comment

Respond

Add Comment

Respond

Add Comment

"After recording all scores, reviewers in the treatment condition proceeded to a screen in which they observed their scores next to artificial scores attributed to other reviewers. 'Other reviewers' were randomly assigned to be described as either scientists with MESH terms like yours or data science researchers...The artificial 'stimulus' scores were presented as a range, e.g. '2-5', and the entire range was randomly assigned to be above or below the initial overall score given by a reviewer. The stimulus scores thus appeared as originating from multiple reviewers (although we did not indicate how many), whose opinions were unanimously different from those of the subjects in the experiment."

Respond

Add Comment

@Ignacio

Respond

Add Comment

I'm shocked, SHOCKED to discover high-status individuals are relatively immune to the opinions of others.

On a more serious note, this is right out of Kahneman.

High status individuals like myself have good reason to not listen to other peoples' opinions. For one, I am rich and Ivy League educated. Two, they are not. That is why they are poor and I am rich.

Respond

Add Comment

Respond

Add Comment

look at the Karen O and Danger Mouse. Popular radio today is mostly mix tapes of popular pop stars. But Karen O, Beck, Phoenix, these people used to be rock stars. What research did they do?

Respond

Add Comment

I wouldn't assume that just because this is a factor in the medical field, that it's a factor in every field.

"covered, concealed, or protected", and "writing".

Respond

Add Comment

Respond

Add Comment

See Delphi estimation technique

Respond

Add Comment

Anybody who ever sat through the Faculty Senate knows this.

Respond

Add Comment

Respond

Add Comment