Growth Mindset Replicates!

A lot of psychological research has failed to replicate, throwing cold water on the entire field. “Grit” and the “growth mindset”, the two taglines of superstar researchers Angela Duckworth and Carol Dweck, checked all the boxes for predictive failure including the requisite TED talks (Duckworth, Dweck), best-selling popular books (Duckworth, Dweck) and genius awards and, to be sure, there has been lots of puffery about the “incredible potential” and “profound impact” of grit and the growth mindset. But, to their great credit, Duckworth and Dweck have taken the replication crisis to heart and have sought to address it. Working with a large team (PI David S Yeager), the authors have tested a growth mindset intervention in 65 randomly chosen schools with over 12,000 students representative of the United States grade 9 population.

Here is what is notable: The analyses were pre-registered, the data were collected by independent researchers and key parts of the model were analyzed by independent statisticians in a blinded dataset.

To achieve arms-length independence, a research firm not involved in designing the materials or study hypotheses drew the sample, recruited schools, facilitated treatment delivery, obtained administrative data, and cleaned and merged data. Data were processed blind to treatment status.

…A random sample of schools, rather than a convenience sample, meant that it represented the full array of the U.S. public educational contexts.

Data were analyzed following a pre-registered analysis plan (the so-called “preregistration challenge,” osf.io/afmb6/) that was developed by an interdisciplinary team, including one external evaluator. All analyses were “intent to treat” (ITT); data were analyzed as long as students saw the first page of the randomized materials.

independent statisticians reproduced the key moderation findings by estimating a hierarchical, nonlinear Bayesian model using a blinded dataset that masked the identities of the variables, to further reduce the possibility of chance findings.

Ok, so what were the results?

Based on administrative records, 9th grade adolescents assigned to the growth mindset
intervention, as compared to the control activity, earned slightly higher GPAs in core classes at
the end of 9
th grade. On a 4-point grade metric (“A” = 4.0, “B” = 3.0, etc.), the average treatment
effect was 0.03 grade points,
SE = .01, N = 12,542 students, k = 65 schools, t = 3.09, P = .003.

In other words, a small, positive effect. But this small effect is coming from a small intervention, two online survey/interventions of 25 minutes each that could be easily scaled to the entire country or even worldwide. We have come a long way from the “mindset revolution” but who am I to discount a marginal revolution? Moreover, the average effect hides heterogeneity, the effect was bigger on the students who needed it most.

as expected, average effects were small because many students
are already doing well, do not have motivational issues, or are not in environments that
encourage or support growth-mindset behaviors. When we take account of such factors, more
noteworthy effects emerge. The improvements in the gateway outcome of 9
th grade GPA were
concentrated among adolescents who are at significant risk for compromised well-being and
economic welfare: those with lower levels of prior achievement attending relatively lower achieving schools. The finding that an intervention can redirect this adolescent outcome in this
sub-group, in under an hour, without training of teachers, and at scale (i.e. in a random sample
of nation’s schools), represents a significant advance.

Overall, this is a very impressive study and one that I suspect will be used to mark the beginning of the post-replication-crisis era.

The ending of the post-replication-crisis era also makes another trend clear–the future of social science will be even more hierarchical and unequal–future social science will be done by large, well-funded teams, run by superstar researchers at top universities. This study, for example, had 10 co-authors from multiple universities and probably cost well over a million dollars. The smaller the effect the bigger the team that will be needed to find it.

Addendum: A big meta-analysis out today also finds very small effects for growth mindset (correlation of growth mindset with achievement=.01) but the effects are probably real especially for academically high-risk students and low-SES students and perhaps they could be magnified by better interventions.

Hat tip: Stuart Richie.

Comments

The methodology might be impressive, but if the conclusion is just " the average treatment effect was 0.03 grade points", I am not sure there is any reason to be excited by this intervention. Probably this has only real benefits for a handful of students? Yes, more vulnerable students, might benefit from ANY intervention, ex .having once per week dinner with a GMU professor for one semester. But is it really worth spending 1 million dollars to study it? I am not sure...

Well, there wouldn't be any reason to be excited if the effect weren't greater for at-risk students. Also, keep in mind that even though the experiment was costly, the intervention itself is very cheap (takes less than an hour of student time and doesn't require teacher training). So...if it's not doing anybody any harm and is providing a benefit to students most at risk of under-performing, why not?

Sorry Alex this is BS.

At first I was unpersuaded by your arguments. Then I thought about it, realized that you didn't even have any arguments and came to the conclusion that I was unpersuaded by you.

The implicit argument is that this fits the profile of things that turn out to be BS.

... Alex & Tyler lately seem to pursue very oddball/splashy topics and speculation. Must be some type of marketing approach to promote/sustain their brand & a general audience

This is not even remotely oddball if you are familiar with the literature

Almost certainly noise. If not noise, then the Hawthorne effect. The Hawthorne effect seems also to be behind the tests showing superiority of the Dvorak keyboard. The people tested on the new keyboard got hours of training on the new keyboard, but the controls did not get hours of training on the old keyboard.

I don't think so. Neither was taught any new study skills. Those randomly assigned to control saw information about the localization of brain functions, and were asked to complete exercises to internalize that topic. Those assigned growth mindset were taught about the brain's neuroplasticity, and asked to do exercises internalizing that topic. Neither should have predictably resulted in better grades ex post.

The preprint pdf is available for download at the link Alex provided, so you could see for yourself. Check out the diagram on page 68.

One of the reasons for this paper's length (its body ends on p. 18, but the document is 74 pages long) appears to be its degree of transparency. I find all the disclosure at the end about the ex ante design and detail on the data pretty remarkable.

Sorry, this is just wishful thinking.

As in, it is wishful thinking that you actually glance at the paper or even read the blog post in full to realize that your concern is of no relevance to this experiment?

Mood affiliation isn't always funny, but I find the circular logic at play in this realm amusing.

People without a growth mindset insist that a growth mindset effect does not exist. By not having a growth mindset, they cannot benefit from its effect.

Somewhere, Joseph Heller is laughing.

I don't see how personally having a growth mindset would in any way affect whether you think growth mindset "works" to improve grades

The whole point of pre-registration, large samples and blinded analysis is to make claims of "noise" implausible. And they are.

Thank God the replication crisis era is now behind us and we are back to True Science.

We should be worried that "the future of social science will be even more hierarchical and unequal", not because of some socialistic, egalitarian beliefs, but because such concentration is bad for science. We need many researchers who are working independently and who have some (but not too much) skin in the game. This future is the opposite. A team of 20 researchers across 6 universities is essentially one "researcher unit", due to groupthink and aversion to contradicting one's collaborators. And with very costly studies, there is a huge incentive to fudge results any way possible (I know it says analysis was independent, but they will find a way..........)

This is the wrong way. Instead, we should be fostering large numbers of independent studies by independent teams. E.g. for a research question X, if 30 teams find a positive effect, 10 no effect and 20 a negative effect you could conclude the effect is probably zero.

Maybe an econometrician can tell us why a single huge study is better than many smaller-scale replications.

It's interesting to compare our attitudes towards clinical trials of drugs to our attitudes towards social science research. I don't hear too many folks clamoring for smaller, more boutique drug trials. And no one seems to think that large drug trials are counterproductive because big teams are prone to "groupthink". Small teams are more likely to sample haphazardly, avoid preregistration, and analyze unblinded data. Saying that they're better seems to amount to a "measure none, cut 30x" attitude.

An alternative to one big study is a many-labs collaboration where many teams of researchers conduct a study according to a commonly agreed-upon plan and the results are then meta-analyzed. The growth mindset effect discovered in this study, for example, is so small that if it had been conducted by say ten teams, each with 1/10 of the sample size, it could have been that none of them would have obtained significant results (without p-hacking). Only by combining the results would the small effect be discovered. There have been several such collaborative replication efforts in psychology recently. This seems like a very good development.

I'm not sure a small team could reasonably do such a high-quality study. You can't replace high-quality studies with any number of low-quality ones.

Quality is no problem for small teams - in fact, the reduced admin loads probably make it easier to achieve. The problem would be that the sample size would (probably) be much smaller.

That's an example of a quality issue that prevents small teams from being as effective as large ones. No number of small-sample studies, no matter how well-performed, will add up to one well-performed large-sample study.

"We need many researchers who are working independently."

But the problem was that lots of independent, small efforts lead to a lot of file-drawer problems and p-hacking (and even outright fraud) and resulted in a lot of garbage. Perhaps one model that may work is to think of independent researchers and small teams as competing to produce candidate hypotheses to be tested in major, expensive studies such as this one.

Exactly. The 30-different-teams approach is not a hypothetical. That's what we already had and what produced the replication crisis. And, your model for independent researchers is correct: they will have to go back to thinking instead of number-crunching, e.g., theory. Maybe, the other social sciences are too soft to allow for meaningful theory, but there are still plenty of unsolved theoretical problems in economics.

Google hit: "Why is it that when some people encounter obstacles to a goal it stops them in their tracks while other people are spurred on by the haters and the challenges life presents?" from "5 Ways to Develop a Growth Mindset Using Grit and Resilience" - Positive Psychology Program.

You're joking AlexT?! This is pop psychology at its worse! haters? Seriously? H8-ers? Call me a silent hater of this stuff.

Of course, a fair question is whether the small gains persist. This appears to happen a lot in education. An intervention appears to have an impact in period X, then the sample is retested at X + Y and the benefit has been eliminated. Given how small the benefit appears to be here, any reduction in benefit due to aging really calls into question growth mindset.

With all of that said, I support growth mindset because it’s hard to see how it can hurt from an educational perspective though I think policymakers need to recognize fixed mindset likely describes the world.

Still, if parents and teachers are giving constant encouragement for students to push themselves, the effects should persist. It could just be a fairly self-evident result that continual encouragement, reward, and praise for effort leads to higher student achievement. Not very surprising really.

That's a valid concern and worthy of additional analysis. I hope someone does follow up on that because the evidence here for growth mindset interventions is pretty good and warrants additional study.

'future social science will be done by large, well-funded teams, run by' ad agencies with large data sets and defined goals. They just won't publish their research.

A brief summary of what constitutes growth mindset would be useful.
I'm going to go google it, but it would be nice if it was addressed in the post.

It's the Left-Wing fantasy that IQ doesn't matter and children can "improve" through effort.

Well, a person who actually studies is going to learn more than a person who doesn't study, no?
IQ is the ability to learn, not the amount of actual learning.

No. It doesn't work like that except in Left-Wing-Land.

So my dad was just born knowing theoretical physics. He didn't have to go to graduate school to get a doctorate to learn it?

But it requires a high IQ. The "growth mindset" BS basically says everyone can improve which isn't remotely true outside of Left-Wing Land.

Everyone can improve what? Their IQ? Or the stuff they actually learn?
I think there's some evidence that brain plasticity allows IQ improvement by a few points. But beyond that, a person with a low IQ can still learn more it just takes longer for them to learn, and their ceiling of concepts they can grasp is lower. Seems to me that growth mindset is about optimizing learning potential, not changing IQ.

It's about left-wing feel goodism.

Hazel, you are proving Subtlety's subtle point by not learning she is trolling. Given a higher IQ, you would be able to recognize the troll.

It's not supposed to just be ambition (the desire to do well), or interest in a topic, it's the idea that you see yourself as having no fixed level of limitations to your ability or knowledge (growth mindset).

Essentially, the idea is that if someone is obsessed with solving problems in maths or learning about physics, but believe that their brain has computational limits of some kind, and that they have a fixed level of intelligence that presents a limitation, they will tend to learn less than someone who believes no such thing. This is in contrast to the idea that interest in a topic and IQ is probably sufficient.

You can easily imagine reasons why a growth mindset might not be an asset - people who believe there are limits to their intelligence and ability may have smarter and more tactical strategies to learn, for instance.

In reality growth vs fixed is probably a scale, if even well described by a non-binary level of simplification, and it's likely that researchers simply gerrymander the cutoffs of what is considered a "growth mindset" (e.g. include the people who have reasonable expectations, don't cutoff at the moronic 10,000 hour rule idiots) until it proves what they want (usually that "openness, diversity and aspirations are good things", them being libs of various sorts).

As opposed to the right-wing fantasy that IQ explains all? Do you really think that effort doesn't have an affect on improvement? Is this something that even needs to be debated? You hear the right lambasting "SJW"s for promoting ideology over reason (100% accurate, imo), it's pretty amazing to see the right doing exactly the same thing.

Thanks for the comments from the Hillary-Bot camp.

Also in right-wing fantasy land, lead poisoning and iron-deficiency anemia have no effect on IQ.

Not relevant to the contemporary US. The people from Flint were low IQ anyway.

But very relevant to sub-saharan africa, where 90% of the babies have iron deficiency anemia.

we are talking about educational policy here in the US. Guess you didn't inherit your father's IQ.

But I thought IQ was genetic!

If anything, the "you can achieve what you want if you belive in yourself and try" seems more a RIGHT-WING fantasy (well, not of the old aristocratic right-wing, , but seems in line with the new pro-capitalism, pro-self-made-man, right-wing(

The working man just needs to TRY HARDER!

Of course, for pro-freedom purposes, the right to do so at $0.01/hour shall not be infringed.

Lots of people try hard for free, should we outlaw that?

Which really illustrates the underlying conflict between the worldview of the alt-right genetic determinists and the libertarians, doesn't it?

If your abilities are determined at birth, then whether you succeed or not is not a matter of free will.

Not really; Libertarians do not systematically cease to believe in inheritance of characteristics, or cease to believe there are constraints on personal growth, or even a deterministic physical law, so how could it? Their liberty thing is not dependent on Horatio Alger-ism. ("Where are the Calvinist Libertarians? Why, right here.").

So Much For Subtlety:

Height is almost completely genetic and extremely predictive of basketball skill, but NBA players still need to practice playing basketball or they will underachieve.

Steve Sailer usually says IQ is about half genes, half environmental. I understand the need to push back against a prevailing narrative that "it's all nurture", but it's pretty obvious that you need to work/study/whatever to achieve your full potential on the environmental side.

A preference for strawman arguments is not a sign if intelligence, it is a sign of lack of respect for your fellow human beings who inhabit the planet.

Specifically, insisting that nurture is of high relevance while refraining from wading into debates with persons whose main agenda is to promote eugenics mindsets is not inconsistent. If it was possible to discuss genetic aspects without attracting a bunch of wannabe de facto genocidalists, then those discussions could more easily happen. But, the moment genes and economic outcomes appear in the same subject matter, all sorts of types come out of the woodworks -- most of whom quite obviously do not put it in such terms -- claiming that some group or another may be deserving of eradication.

For example, identifying some genetic role in differential learning abilities could in the future enable to tailor education strategies. But we cannot have that discussion because it'll immediately degrade into proto-Nazi argumentation.

This does kind of sum up the problem. It would be nice to be able to discuss genetic variances in IQ and the like, but if that stuff gets out into the mainstream, there are a lot of below-average-IQ people out there who will not understand it and will start thinking that their genes make them inherently superior to others.

Troll Me said:
If it was possible to discuss genetic aspects without attracting a bunch of wannabe de facto genocidalists.

That's pretty dark accusation Troll Me, can you provide recent statements made by genocidalists?
Perhaps what you think you read was just a voice in your head?

I'd like to see genetic science enhancing everyone to posses the innate capacity of a celestial physicist, as Johannes Kepler referred to his hobby.

"If it was possible to discuss genetic aspects without attracting a bunch of wannabe de facto genocidalists."

Translated for honesty: "I didn't understand this factual thing, rejected it due to various emotion commitments, and slandered people who actually did understand it. Now it's becoming increasing difficult to do so, so I must double down on talking darkly about Nazis and deplorables to maintain my preferred equilibrium".

If the preferred equilibriumthat is breaking down, is not systematically mistreating people based purely on their genetic phenotype, what the hell is wrong with that? Do you want to live in a society which is fragmented into ethnic tribes that hate each other?

The only reason it is predictive is that you define "skill" as being in the top 1% of all players. Genius may have a strong genetic component, but below that height and other methods play far less of a sorting factor than people think. A willingness to learn will be far more effective than genetic determinism for virtually anything below that.

Very possibly, trying in situations that matter is more important for working life than results on a test that has precisely zero impact on anything that matters.

There is also the risk that people who perform well on a standardized test that has zero impact on anything that matters may thereafter delude themselves into believing that they are broadly superior to others, an attribute which tends to degrade the ability to function effectively as a member of a team.

For example, people like yourself who troll people on such questions are probably not welcome on very many teams, since regardless of whatever potential underlying intelligence you'd probably have a less than ZMP effect on team outputs due to inefficiently frequent but justified application of middle fingers and the like.

It must be comforting to think so

Hmmm... can you point to something we can measure that cannot be improved through effort?

Ha. Yes, liberal Fantasy that hard work pays off.

Google it? Why not look at one of the many links that Alex gave. Watch one of the TED talks, for example.

Don't read books kids! The cards have already been dealt... - So Much for Subtlety's educational poster

No need to provide tax-payer funded subsides for individuals with less than 125 IQ is my educational policy.

How's the view from the doghouse?

Anyone who believes it's impossible that they'll become Terrance Tao with just enough effort on maths = someone who doesn't believe they are capable of remembering the contents of the newspaper they read this morning - Jeff's stance on the Growth Mindset

I think by reducing the growth mindset to "anyone can be anything" you're either being intellectually dishonest or you just don't actually understand the ideas behind the theory. To be fair, a lot of proponents of growth mindset have also misconstrued simplified the ideas into "just work hard and you can do anything", so this sort misconception is understandable (and Carol Dweck has actually written quite a bit about how the theory is being misapplied in school-settings).

One of the initial experiments that Dweck performed was looking at how a person's mindset affected their ability to persevere when given a challenging task. What she found was that kids who were praised for effort rather than for intelligence were much more likely to try different methods and to work for a longer period of time when given a challenging task. They were also much more likely to attempt a challenging task when given the choice between different tasks, whereas the group who were praised for intelligence were much more likely to attempt the easy task. Dweck termed this as having a "fixed-mindset". Her theory was that when faced with challenges, those that believed they were intelligent and that this intelligence was not malleable would shirk challenges because failing at challenges would mean that they "just weren't as smart as they thought they were or that people had been telling them they were".

I think the more controversial part of her theory is that we can in effect "change" a person's mindset. My opinion is this, if by having what she calls a "growth mindset" allows a person to persevere when faced with challenging situations, is it really so terrible to try to think about ways to instill this type of mindset in our children?

This is total bullshit!

One might conclude that most (nearly all?) social science research done by small teams, with small data sets, etc. is going to be of very limited (essentially no?) value. There will probably be exceptions (the social science equivalent of the discovery of penicillin), but not many.

This would imply that most of the current social science work is somewhere between worthless and fraudulent. Should funding for such work be curtailed, at the university and grant making institutions? Should "Direction State" faculty focus on teaching and stop pretending that they are doing useful research?

Datasets need to be representative, not large, to obtain robust results. A single large dataset, such as from some basic information from a census, should be sufficient to ensure representativity of small datasets.

Insisting that large datasets are an a priori condition for quality social sciences research is basically an argument in favour of excessive data collection, which may be all-too-easily re-purposed toward anti-freedom ends.

So, broadly, there was some truth in the attitude of old-fashioned teachers. What the dim pupils needed was a figurative kick up the arse. Mind you "two online survey/interventions of 25 minutes each" seems likely to be rather more expensive than the figurative kick.

Positive reinforcement and promoting self confidence in one's ability to grow and thrive need not be viewed as mutually exclusive with enforcement of discipline.

"One might conclude that most (nearly all?) social science research done by small teams, with small data sets, etc. is going to be of very limited (essentially no?) value."

Keep in mind that the original growth mindset research (e.g. the work being replicated by this large study) was done by small teams with small data sets. The hypotheses to test with expensive, large-scale studies have to come from somewhere.

buried your lede: "The smaller the effect the bigger the team that will be needed to find it."

I applaud the effort. But the effect size strongly suggests that previous findings by Duckworth et al in (I assume) vastly smaller samples really were driven mostly by noise. And studies of this quality will not be the norm any time soon; Duckworth and her graduate students will in all probability proceed to publish dozens of small studies that have been wrangled into giving up a p<0.05, despite the fact that this study suggests large samples are necessary to accurately measure the effects.

The worst-case scenario is that this study is used as a cover for sloppy research in the future -- one minimal finding replicated, so therefore the entire literature, with all it's inflated effect sizes and contradictory interactions and subgroup effects, need not be scrutinized! The fact that a non-zero effect exists does not mean that the bulk of the literature is actually measuring that effect.

In other words, a small, positive effect. But this small effect is coming from a small intervention, two online survey/interventions of 25 minutes each that could be easily scaled to the entire country or even worldwide. We have come a long way from the “mindset revolution” but who am I to discount a marginal revolution? Moreover, the average effect hides heterogeneity, the effect was bigger on the students who needed it most.

I don't know if I'd call this confirmation. Grit and resilience were always sold as a *large* positive effect. If you go looking for a large effect and find one which is barely distinguishable from zero, are you really confirming the theory.

When I was teaching, the summed score for all assignments in the semester often came to about a thousand points. Call it 100 points per grade point, so that three hundredths of a grade point is three points. A multiple choice question was worth 5 points, so the size of this effect is about the same as knowing the answer versus guessing for one multiple choice question on one test.

Putting it another way: if we had tried to boost participation by giving out five bonus points on one assignment for filling out a brief online questionnaire about grit and resilience, those five points would be about twice the size of the effect they claim to measure! I think the only fair description of these results is to say there was no effect.

Remember, the statistician's fallacy is that all errors are random and statistical. The real world has sources of error which are not accounted for in the statistical error budget.

N=12,542. That's a very big N and as N increases the chance of a spurious statistically significant finding increases right along with it. Given that and the microscopic effect size I'd hardly declare victory after a single replication. But it'll sell books just like the next statistically significant pancreatic cancer break though that extends survival by an average of 2.7 days (which is what this effect size would map to for stage IV pancreatic cancer) will sell lots of pills. When the Elois run out of money to buy all the B.S. we sell them that's when they go on the menu.

Doesn't law of large numbers mean large N decreases the chance of a spurious finding?

No. It’s counter-intuitive but here’s something that may help you: http://onlinelibrary.wiley.com/store/10.1111/cts.12178/asset/cts12178.pdf;jsessionid=091A3C626535680B5920BF8CC0BEE69D.f01t02?v=1&t=jei9cmuq&s=92480d2d24f2725efd5267747331c2eaacf433e4

How large was the effect on the weaker students? 0.03 grade points is not a replication. At that sample size, pretty much any difference will be statistically significant.

It took me some time before I realised all Dweck did was to just confirm the fluid intelligence / crystallised intelligence theory. It doesn't necessarily make students smarter - the cognitive limit is still there, and if you run the students through psychometric (IQ) tests like Raven's Progressive Matrices they'll still end up with the same scores (provided you change up the patterns). No matter how hard they believe in the Growth Mindset, their fluid intelligence is fixed.

Of course, fluid intelligence is useless without any inputs to process into crystallised experience. All the Growth Mindset does is to focus on the conscientious aspect of personality and work on saturating the fluid intelligence portion with inputs to maximise gains in crystallised intelligence.

So the students do better on academic tests which are mainly tests of crystallised intelligence. That's great, but let's be realistic about what it can do, what it cannot do, and why.

For various reasons, there's also a downside to pushing students too far towards conscientiousness if the Growth Mindset really does affect this personality trait. In Singapore, I have a strong feeling that all the rules and bureaucracy are the result of hard-working, well-meaning and conscientious rule-followers in the public sector thinking of ever more rules for people to follow.

Comments for this post are closed