A lot of psychological research has failed to replicate, throwing cold water on the entire field. “Grit” and the “growth mindset”, the two taglines of superstar researchers Angela Duckworth and Carol Dweck, checked all the boxes for predictive failure including the requisite TED talks (Duckworth, Dweck), best-selling popular books (Duckworth, Dweck) and genius awards and, to be sure, there has been lots of puffery about the “incredible potential” and “profound impact” of grit and the growth mindset. But, to their great credit, Duckworth and Dweck have taken the replication crisis to heart and have sought to address it. Working with a large team (PI David S Yeager), the authors have tested a growth mindset intervention in 65 randomly chosen schools with over 12,000 students representative of the United States grade 9 population.
Here is what is notable: The analyses were pre-registered, the data were collected by independent researchers and key parts of the model were analyzed by independent statisticians in a blinded dataset.
To achieve arms-length independence, a research firm not involved in designing the materials or study hypotheses drew the sample, recruited schools, facilitated treatment delivery, obtained administrative data, and cleaned and merged data. Data were processed blind to treatment status.
…A random sample of schools, rather than a convenience sample, meant that it represented the full array of the U.S. public educational contexts.
… Data were analyzed following a pre-registered analysis plan (the so-called “preregistration challenge,” osf.io/afmb6/) that was developed by an interdisciplinary team, including one external evaluator. All analyses were “intent to treat” (ITT); data were analyzed as long as students saw the first page of the randomized materials.
… independent statisticians reproduced the key moderation findings by estimating a hierarchical, nonlinear Bayesian model using a blinded dataset that masked the identities of the variables, to further reduce the possibility of chance findings.
Ok, so what were the results?
Based on administrative records, 9th grade adolescents assigned to the growth mindset
intervention, as compared to the control activity, earned slightly higher GPAs in core classes at
the end of 9th grade. On a 4-point grade metric (“A” = 4.0, “B” = 3.0, etc.), the average treatment
effect was 0.03 grade points, SE = .01, N = 12,542 students, k = 65 schools, t = 3.09, P = .003.
In other words, a small, positive effect. But this small effect is coming from a small intervention, two online survey/interventions of 25 minutes each that could be easily scaled to the entire country or even worldwide. We have come a long way from the “mindset revolution” but who am I to discount a marginal revolution? Moreover, the average effect hides heterogeneity, the effect was bigger on the students who needed it most.
…as expected, average effects were small because many students
are already doing well, do not have motivational issues, or are not in environments that
encourage or support growth-mindset behaviors. When we take account of such factors, more
noteworthy effects emerge. The improvements in the gateway outcome of 9th grade GPA were
concentrated among adolescents who are at significant risk for compromised well-being and
economic welfare: those with lower levels of prior achievement attending relatively lower achieving schools. The finding that an intervention can redirect this adolescent outcome in this
sub-group, in under an hour, without training of teachers, and at scale (i.e. in a random sample
of nation’s schools), represents a significant advance.
Overall, this is a very impressive study and one that I suspect will be used to mark the beginning of the post-replication-crisis era.
The ending of the post-replication-crisis era also makes another trend clear–the future of social science will be even more hierarchical and unequal–future social science will be done by large, well-funded teams, run by superstar researchers at top universities. This study, for example, had 10 co-authors from multiple universities and probably cost well over a million dollars. The smaller the effect the bigger the team that will be needed to find it.
Addendum: A big meta-analysis out today also finds very small effects for growth mindset (correlation of growth mindset with achievement=.01) but the effects are probably real especially for academically high-risk students and low-SES students and perhaps they could be magnified by better interventions.
Hat tip: Stuart Richie.