Why isn’t “stereotype threat” stronger in the data?

by on November 29, 2017 at 12:44 am in Data Source, Education, Science | Permalink

From a recent survey by Pennington, Heim, Levy, and Larkin:

This systematic literature review appraises critically the mediating variables of stereotype threat. A bibliographic search was conducted across electronic databases between 1995 and 2015. The search identified 45 experiments from 38 articles and 17 unique proposed mediators that were categorized into affective/subjective (n = 6), cognitive (n = 7) and motivational mechanisms (n = 4). Empirical support was accrued for mediators such as anxiety, negative thinking, and mind-wandering, which are suggested to co-opt working memory resources under stereotype threat. Other research points to the assertion that stereotype threatened individuals may be motivated to disconfirm negative stereotypes, which can have a paradoxical effect of hampering performance. However, stereotype threat appears to affect diverse social groups in different ways, with no one mediator providing unequivocal empirical support. Underpinned by the multi-threat framework, the discussion postulates that different forms of stereotype threat may be mediated by distinct mechanisms.

Or from Wikipedia:

Whether the effect occurs at all has also been questioned, with researchers failing to replicate the finding. Flore and Wicherts concluded the reported effect is small, but also that the field is inflated by publication bias. They argue that, correcting for this, the most likely true effect size is near zero (see meta-analytic plot, highlighting both the restriction of large effect to low-powered studies, and the plot asymmetry which occurs when publication bias is active).[

Earlier meta-analyses reached similar conclusions. For instance, Ganley et al. (2013)[10] examined stereotype threat on mathematics test performance. They report a series of 3 studies, with a total sample of 931 students. These included both childhood and adolescent subjects and three activation methods, ranging from implicit to explicit. While they found some evidence of gender differences in math, these occurred regardless of stereotype threat. Importantly, they found “no evidence that the mathematics performance of school-age girls was impacted by stereotype threat”. In addition, they report that evidence for stereotype threat in children appears to be subject to publication bias. The literature may reflect selective publication of false-positive effects in underpowered studies, where large, well-controlled studies find smaller or non-significant effects:

Personally, I find stereotype threat to be a very intuitive idea with a fair amount of anecdotal support.  So why aren’t these meta-results more convincing?

1 JJB November 29, 2017 at 12:59 am

A relevant anecdote (from: https://www.richmondfed.org/~/media/richmondfedorg/publications/research/region_focus/2012/q2-3/pdf/interview.pdf):

RF: Your paper with Roland Fryer and Steven Levitt
came to a somewhat ambiguous conclusion about
whether stereotype threat exists. But do you have a
hunch regarding the answer to that question based on
the results of your experiment?

List: I believe in priming. Psychologists have shown us the
power of priming, and stereotype threat is an interesting
type of priming. Claude Steele, a psychologist at Stanford,
popularized the term stereotype threat. He had people
taking a math exam, for example, jot down whether they
were male or female on top of their exams, and he found that
when you wrote down that you were female, you performed
less well than if you did not write down that you were female.
They call this the stereotype threat. My first instinct was
that effect probably does happen, but you could use incentives
to make it go away. And what I mean by that is, if the
test is important enough or if you overlaid monetary incentives
on that test, then the stereotype threat would largely
disappear, or become economically irrelevant.

So we designed the experiment to test that, and we found
that we could not even induce stereotype threat. We did
everything we could to try to get it. We announced to them,
“Women do not perform as well as men on this test and we
want you now to put your gender on the top of the test.”
And other social scientists would say, that’s crazy — if you do
that, you will get stereotype threat every time. But we still
didn’t get it.

What that led me to believe is that, while I think that
priming works, I think that stereotype threat has a lot of
important boundaries that severely limit its generalizability.
I think what has happened is, a few people found this result
early on and now there’s publication bias. But when you talk
behind the scenes to people in the profession, they have a
hard time finding it. So what do they do in that case? A lot of
people just shelve that experiment; they say it must be
wrong because there are 10 papers in the literature that
find it. Well, if there have been 200 studies that try to find
it, 10 should find it, right? This is a Type II error but people
still believe in the theory of stereotype threat. I think that
there are a lot of reasons why it does not occur. So while I
believe in priming, I am not convinced that stereotype
threat is important.

Reply

2 So Much For Subtlety November 29, 2017 at 4:40 am

Well, if there have been 200 studies that try to find it, 10 should find it, right? This is a Type II error but people still believe in the theory of stereotype threat. I think that there are a lot of reasons why it does not occur. So while I believe in priming, I am not convinced that stereotype threat is important.

The Soviet-style in science – they did not find evidence of witches, but witches must exist nonetheless.

The obvious answer is that pretty much everything in this field is made up by the sub-standard politically motivated buffoons who populate the modern university world. You’re better off believing the exact opposite of whatever they tell you. In the modern West, growing up means realizing that your college teacher was wrong about everything and your grandmother pretty much right.

Reply

3 JWatts November 29, 2017 at 9:00 am

“…growing up means realizing that your college teacher was wrong about everything and your grandmother pretty much right.”

No, that’s not correct. My grandmother was wrong about plenty. However, she wasn’t nearly as knowledgeable as my college teachers. So, her ignorance kept her from being wrong about as many diverse subjects. And her humility meant that she wasn’t absolutely convinced she was right about anything in particular, even when the evidence was clearly to the contrary.

Reply

4 peri November 29, 2017 at 10:32 am

Thanks to your grandmother comment, I’m reminded of all the ways the old woman could tell Huck wasn’t a girl.

“Well, try to remember it, George. Don’t forget and tell me it’s Elexander before you go, and then get out by saying it’s George Elexander when I catch you. And don’t go about women in that old calico. You do a girl tolerable poor, but you might fool men, maybe. Bless you, child, when you set out to thread a needle don’t hold the thread still and fetch the needle up to it; hold the needle still and poke the thread at it; that’s the way a woman most always does, but a man always does t’other way. And when you throw at a rat or anything, hitch yourself up a tiptoe and fetch your hand up over your head as awkward as you can, and miss your rat about six or seven foot. Throw stiff-armed from the shoulder, like there was a pivot there for it to turn on, like a girl; not from the wrist and elbow, with your arm out to one side, like a boy. And, mind you, when a girl tries to catch anything in her lap she throws her knees apart; she don’t clap them together, the way you did when you catched the lump of lead. Why, I spotted you for a boy when you was threading the needle; and I contrived the other things just to make certain…”

Reply

5 Art Deco November 29, 2017 at 11:39 am

In the modern West, growing up means realizing that your college teacher was wrong about everything and your grandmother pretty much right.

I’m not sure my college teachers had any subjects of conversation which intersected with my grandmother’s table talk.

Reply

6 Boonton November 29, 2017 at 1:33 pm

You’re better off believing the exact opposite of whatever they tell you. In the modern West, growing up means realizing that your college teacher was wrong about everything and your grandmother pretty much right.

Your grandmother, though, probably made few or no decisions based on standardized test results. So if you took this seriously you would not only toss out the ‘progressive theory’ here about implicit bias but also the meritocratic theory about using ‘objective measures’ to make decisions like job hiring or school admissions.

If people who speak like ‘So much for…’ actually thought this out and advocated it, they would have a very interesting argument. Instead they often revert to a half measure that’s actually less than the best that could be given to us by either college professors or grandma.

Reply

7 TGGP November 29, 2017 at 8:07 am

Of course, now all the priming literature is hard to take seriously.

Reply

8 Eric Rasmusen November 29, 2017 at 8:36 am

Not because of the List finding. His finding is that this stereotype threat has so little effect it can’t be primed, like priming people by saying “Today the Martians are going to land in our city, so you are going to be nervous and do badly on this test. And you will commit suicide an hour after it ends.” Priming, like hypnotism, doesn’t work for everything. The subject has to be ready to follow.

Reply

9 Purple Mutt November 29, 2017 at 1:05 am

If I’m under “stereotype threat,” I might work less conscientiously, or disengage more easily when encountering obstacles.

But I don’t think it would affect my performance on short max-effort tests.

Basically: I agree that this concept feels so natural that it probably exists, but these researchers are looking for it in the wrong places.

Reply

10 Purple Mutt November 29, 2017 at 1:07 am

Clarification: it might reduce test performance, but primarily by reducing accumulated quantity of practice. If the stereotype threat is only applied during the test, I wouldn’t expect much effect.

Reply

11 Petar November 29, 2017 at 9:43 am

+1, as they say

Reply

12 ʕ•ᴥ•ʔ November 29, 2017 at 7:17 am

Stereotype threat might also produce “no shows” which would not appear in performance studies.

But I think the real problem with the meta survey is that they did look for a common effect across “diverse social groups [working] in different ways”

Human interactions are similar but not that similar.

Reply

13 static November 29, 2017 at 9:26 am

It would not produce “no shows” because the threat in the experiment is not delivered until after they have “shown”. If the threat is delivered before they arrive for the test, that would be a different environmental context and that part of the data that would be measured.

Reply

14 ʕ•ᴥ•ʔ November 29, 2017 at 9:32 am

Right. I was speaking of the wider social impact of something like “stereotype threat” .. who takes SATs, who does math homework, etc.

The problem with constrained studies, especially of exam setting, is that they are very constrained.

Reply

15 static November 29, 2017 at 9:24 am

Stereotype presumes a certain type of reaction. It is possible that there are lots people with a more contrarian mindset for whom telling them that the mean score for people in a group they belong to is low try extra hard to beat it, the threat becomes motivation. In sports, when a manager puts some comment that denigrates the team on the wall of the locker room, it is a motivational technique to prove those people wrong.

It is also possible that people recognize that intra-group variation is usually greater than inter-group variation, so knowing the average score for group x is lower than the average score for group y does not give any useful information as to their own performance.

Stereotype threat is more believable as a rationalization or coping mechanism after a poor performance or negative result of some kind.

Reply

16 derek November 29, 2017 at 11:00 am

They are doing it wrong.

This test is really hard for women so we add 10 points to your score. Men don’t get it.

I would bet the women would score lower.

Reply

17 Jeff R November 29, 2017 at 11:10 am

Personally, I do not find the idea intuitive. People are not defined by what groups they belong to in their own mind. They think of themselves as individuals, and it seems unlikely that they’d unconsciously be discouraged by stereotypes.

Reply

18 Boonton November 29, 2017 at 1:52 pm

“People are not defined by what groups they belong to in their own mind. They think of themselves as individuals”

Then where exactly do groups exist? Like what exactly is “women” or “white people” or “black people” if no one thinks in their mind “I am/am not a woman/black/white etc.”?

Reply

19 Jeff R November 29, 2017 at 3:59 pm

Bad wording on my part. What I was trying to say was that people think of themselves more as individuals than as members of a group. Their self-esteem, motivation, determination, etc., are products of personal ambitions and goals, rather than any sort of group membership. E.G., no 19-year old gal studies organic chemistry with the goal of raising the status women in science or some BS like that; they do it ’cause they want to get paid.

Reply

20 Boonton November 29, 2017 at 4:21 pm

I agree with that, I also agree groups as such are social constructs but they have objective, measurable impact.

Turn away from politics and consider marketing. When someone says something like “these products sell to millennials while these other products sell to Baby Boomers” that doesn’t mean any 20 year old is shopping in Walmart with the idea that she is carrying a banner for millennials while some 70 year old is shopping on behalf of ‘his group’. Nonetheless, these groups are ‘visible’ in the data even when there’s no actual ‘millennial’ DNA.

21 Dr. Franken November 29, 2017 at 1:11 am

Stereotype threat mainly affects medical tests–like testing for HIV, people under stereotype threat are more likely to test false-positive.

Reply

22 JWatts November 29, 2017 at 9:24 am

Or as another example, people who are treated like monsters, behave like monsters.

Reply

23 Wonks Anonymous November 29, 2017 at 10:05 am

Is this a joke? Hard to tell just reading text on the internet.

Reply

24 Raf November 29, 2017 at 2:31 am

There are different “intuitive” ways in which highlighting a stereotype may affect—or fail to affect–a person during a test, so there is no reason to expect a particular overall result. For example, one person may get nervous when a negative stereotype is highlighted (“oh no, I must be bad too!”), another person may feel more relaxed (“I probably won’t disappoint anyone, since expectations are low”), and yet another person may simply remain indifferent (for a variety of possible reasons).

Reply

25 Jeff L November 29, 2017 at 2:48 am

It has been an amazingly useful tool for the PC narrative and thus is brought up whenever relevant. It is intuitive and there are anecdotal stories in part because the idea has been incessantly promoted. (And the secondary core idea, that insulting someone harms immediate performance, is intuitive but also not clearly correct in all relevant circumstances)

Is this blog post an example of esoteric writing?

Reply

26 Tanturn November 29, 2017 at 10:38 am

Tyler’s basically saying, “I know I’m supposed to believe in this, but I don’t.”

Reply

27 JustChecking November 29, 2017 at 3:21 am

This is all in reference to Claude Steele recently saying (on a podcast called Radiolab) that he will continue to believe in stereotype threat even if the scientific evidence shows otherwise, right?

Reply

28 Purple Mutt November 29, 2017 at 5:38 am

that was my first thought. Great podcast.

I think Alex just posted a blog about replication in criminal economics and how most doest hold up. We should expect more from psychology.

Clearly it is intuitive but how much of an impact can it have during the exam versus its impact during the preparation for the exam is the real question. You should see these threats teased out a bit more by other classifications as well. That is, black men should do better than black women in math, etc..

Reply

29 Anonymous November 29, 2017 at 1:32 pm

If you have to put “clearly” then you know it’s not clear. Maybe if someone was trying really hard to enrage me right before a test they could succeed in discombobulating me enough that I would do more poorly than usual, particularly if it was a writing test. I don’t know.

Reply

30 Lee Wang November 29, 2017 at 4:05 am

Boys boys boys have you learned nothing on Cowen’s blog?

Straussian reading.

Reply

31 Dan November 29, 2017 at 8:43 am

ding ding ding

Reply

32 albatross November 29, 2017 at 4:27 pm

+1

Reply

33 ʕ•ᴥ•ʔ November 29, 2017 at 7:25 am

When less than median GPA white boys avoid college, and learn to rant against “ivory tower liberal elites,” what part is stereotype threat?

Reply

34 TMC November 29, 2017 at 12:26 pm

Less than median GPA white boys don’t pay any attention to ivory tower types. It’s the higher than median GPA white boys who notice their BS.

Reply

35 honeyoak November 29, 2017 at 7:26 am

Am I the only one impressed by the quality of academic writing on Wikipedia? The obsolescence of journals may be closer than we think…

Reply

36 Floccina November 29, 2017 at 12:05 pm

+1

Reply

37 rayward November 29, 2017 at 7:39 am

Doesn’t stereotype threat include sexual harassment? The term is defined in Wikipedia as a situational predicament in which people are or feel themselves to be at risk of conforming to stereotypes about their social group. An article in the WP yesterday reported on alleged sexual harassment by an older, highly-regarded, male creative writing professor. What struck me is that he has allegedly harassed his female students for decades, yet little or nothing came of it. And these aren’t just regular female college students, but very talented writers who went on to successful careers as writers – only students who have shown promise as writers are even admitted into the creative writing program. Now, after Weinstein et al., it seems all of this professor’s former female students are coming forward with tales of sexual harassment, including tales of diminished performance in the creative writing program as the result of the stereotype threat they experienced. Did these women, did all women who have recently come forward with allegations of sexual harassment, feel themselves to be at risk of conforming to stereotypes when they were victimized by their male sexual harassers? If that’s the case, wouldn’t it mitigate the offense committed by the harasser? They asked for it, didn’t they?

Reply

38 JWatts November 29, 2017 at 9:29 am

“Now, after Weinstein et al., it seems all of this professor’s former female students are coming forward with tales of sexual harassment, including tales of diminished performance in the creative writing program as the result of the stereotype threat they experienced.”

That doesn’t seem to be a “stereotype” threat. People who are harassed by someone in direct authority over them are actually being threatened.

Reply

39 albatross November 29, 2017 at 4:28 pm

The most serious stuff Weinstein is accused of isn’t sexual harrassment, it’s rape or sexual assault. Those are massively different things, and putting them all under one category just makes it harder to think straight about the matter.

Reply

40 Matt November 29, 2017 at 8:18 am

“Personally, I find stereotype threat to be a very intuitive idea with a fair amount of anecdotal support. So why aren’t these meta-results more convincing?”

Can’t speak to the specific issue, but “intuitive” and “anecdotal support” are often the quickest and easiest methods for coming to the wrong conclusion.

Reply

41 JWatts November 29, 2017 at 9:33 am

I think that’s why several have mentioned the Straussian reading. That perhaps Tyler knows that intuition and anecdotal support don’t trump actual facts.

Reply

42 Eric Rasmusen November 29, 2017 at 8:38 am

“I find stereotype threat to be a very intuitive idea with a fair amount of anecdotal support.”

Just the sort of situation where empirical work can be useful. And, of course, also just the sort of situation where one ought not to be believe just any old academic papers, because of the limitations of experiments and regressions.

Reply

43 Niroscience November 29, 2017 at 8:44 am

Priming makes sense in a way that living with expectations (high or low) can take a marginal toll.

Whether or not that difference is large enough or common enough for observe statistically is another question.

Reply

44 Bluephy November 29, 2017 at 10:11 am

The intuitive mechanisms don’t apply to one time tasks but to repeated tasks where practice matters and I am doing constrained optimization of how I spend my efforts.

If the stereotype influences my estimate of the likely return on additional practice … to the extent that I believe I might be organically bad at this in a way that practice can’t help or that limits the value of practice … then I practice less.

As a 5’4″ white guy, I’m spending less time in the gym shooting hoops than others, and this might be a rational decision depending on what I am doing instead.

Reply

45 Giorgio November 29, 2017 at 1:41 pm

Stereotype Threat + “The Unbearable Accuracy of Stereotypes” = People are more rational than we thought?

Reply

46 albatross November 29, 2017 at 4:34 pm

I think ST refers specifically to a depression in performance based on being reminded that there’s a stereotype that says people like you do it badly. So if you give a bunch of high school girls a math test, but remind them all beforehand that girls usually don’t do as well as boys at math, that’s a place where you’d imagine ST working. I don’t think ST refers to any upstream causes of this, like Bluephy is proposing, though you could imagine such an effect existing.

Priming and ST are both theories that sort-of intuitively sound like they could be true–it’s not so hard to see the mechanism by which they might work. But the original experiments that demonstrated them look to have been flawed, and later experiments don’t generally find the claimed effects. So probably they’re just wrong–after all, it’s easy to come up with all kinds of plausible theories about this sort of thing, and probably most of them are wrong.

On one side, it’s easy to bash social psychology for not catching these problems earlier. On the other, it’s really good that, as a field, they’ve realized there’s a problem and they’re actually digging around trying to find what has gone wrong in the past.

Reply

47 Bob Knaus November 29, 2017 at 10:21 am

What about an intuitive and anecdotal “Stereotype Motivation”? Seems to me that the number of people daunted by their stereotype might be more or less balanced by those who say “Dammit I’ll show them!”
Gumption, in other words.

Reply

48 Dan in Euroland November 29, 2017 at 10:30 am

My thoughts exactly. I find rebellion (doing the opposite) to be quite prevalent especially in the young.

Reply

49 Boonton November 29, 2017 at 1:44 pm

The problem with the “I’ll show them” mindset is that it’s all well and good for tough talk at a bar but when you’re by yourself taking a test and hitting a tough question, personal doubts about your knowledge seem easier to start haunting you than a cocky arrogance that helps you “fake it to you make it”.

Reply

50 Anonymous November 30, 2017 at 1:32 am

“When you’re by yourself taking a test and hitting a tough question, personal doubts about your knowledge seem easier to start haunting you than a cocky arrogance that helps you “fake it to you make it”.”

I’m looking forward to your study coming out. What would even make you think that you knew this? I’m truly puzzled by statements like this.

Reply

51 Boonton November 30, 2017 at 5:59 am

From what I understand the stereotype threat has only been measured when the test is relatively hard. On easy tests both groups do equally as well.

Presumably a negative prime (“women tend to do worse on this test but everyone try your best”) needs to work something that confirms it (“two hard questions in a row, I’m in over my head on this test”).

Arrogance as a response would mean someone would say something like “two hard questions in a row, now I’m even more pissed that guy in the white coat is really going to shit a brick when he sees how high this woman scores!”. Is that type of response possible? I’m sure but it doesn’t strike me as very likely. Whenever I’ve encountered hard question after hard question on a test I’ve noticed it has a negative impact on my self-confidence.

Something else….I think the arrogance response works best if you’re looking at a team project where you can project “we’re going to win” to your teammates and they can reflect that back to you. “I’m going to beat this test” simply doesn’t work as well IMO. There’s a reason coaches do prep talks for teams all the time but do they work as well for solo competitors? Not so sure.

Reply

52 Brian Donohue November 29, 2017 at 10:33 am

“Personally, I find stereotype threat to be a very intuitive idea with a fair amount of anecdotal support. So why aren’t these meta-results more convincing?”

I don’t. Do others see this as intuitive or anecdotally supported? If so, I’m interested to hear some of these anecdotes.

Regardless, the framing of Tyler’s closing paragraph is pretty much the antithesis of a scientific mindset and a good example of why so much social “science” is suspect and how the social psychologists found themselves in their current mess.

Reply

53 Brian Donohue November 29, 2017 at 10:36 am

Guh. It’s too early in the morning for Strauss. At this point, Tyler is treating the comments section like a bunch of lab rats. Which maybe he should.

Reply

54 Floccina November 29, 2017 at 11:48 am

So white men can jump?

Reply

55 albatross November 29, 2017 at 4:35 pm

…unless you tell us we can’t.

Reply

56 Boonton November 29, 2017 at 1:29 pm

I think the problem here is the assumption priming is not in itself stable.

Let’s say we had this statement: Study X demonstrates if you associate a brand with a popular movie figure, sales will increase.

Evidence: In 1984 a cereal company put Luke Skywalker on their boxes and sales increased 20%.

In 2017 we attempt to replicate by putting Luke Skywalker back on a cereal box and nothing happened to sales. Is the theory false?

Possibly but more likely popular movie figures come and go so the set of ‘positive primes’ and ‘negative primes’ may change while priming as a concept remains valid. In a ‘Black Lives Matter’ and ‘Resist’ world, a prime that Blacks or women do worse on a particular test might be seen as a challenge which actually causes positive performance whereas two decades before when Murray’s Bell Curve as all the rage it had a negative effect.

Reply

57 albatross November 29, 2017 at 4:37 pm

Maybe, but it’s also quite possible that the original studies were just sloppy, or that they just got unlucky and by chance got what looked like a real effect, but was actually just luck.

Reply

58 Boonton November 29, 2017 at 7:21 pm

Let’s say the effect is real, that doesn’t change the fact that stereotypes change over time. I could imagine an Englishman in 1920 talking about ‘Papists’ in regards to Catholics and in 2020 many would just scratch their head at the word. So between point A and B was the effect a real one that vanished or did the effect never exist?

We could do multiple studies at the same moment to tease out whether the effect is due to luck or sloppy controls but even then how does this help unless we also have some model for how the effect will change over time.

Reply

59 Andy November 29, 2017 at 9:41 pm

…becase stereotype threat doesn’t exist?

Reply

Leave a Comment

Previous post:

Next post: