Did Bill Gates waste a billion dollars because he failed to understand the formula for the standard deviation of the mean? Howard Wainer makes the case in the entertaining Picturing the Uncertain World (first chapter with the Gates story free here). The Gates Foundation certainly spent a lot of money, along with many others, pushing for smaller schools and a lot of the push came because people jumped to the wrong conclusion when they discovered that the smallest schools were consistently among the best performing schools.
The chart at left, for example, shows by size the percentage of schools in North Carolina which were ever ranked in the top 25 of schools for performance. Notice that nearly 30% of the smallest decile (10%) of schools were in the top 25 at some point during 1997-2000 but only 1.2% of the schools in the largest decile ever made the top 25.
Seeing this data many people concluded that small schools were better and so they began to push to build smaller schools and break up larger schools. Can you see the problem?
The problem is that because small school don’t have a lot of students, scores are much more variable. If for random reasons a few geniuses happen to enroll one year in a small school scores jump up and if a few extra dullards enroll the next year scores fall.
Thus, for purely random reasons we would expect small schools to be among the best performing schools in any givenyear. Of course we would also expect small schools to be among the worst performing schools in any given year! And in fact, once we look at all the data this is exactly what we see. The figure below shows changes in fourth grade math scores against school size. Note that small schools have more variable scores but there is no evidence at all that scores on average decrease with school size.
States like North Carolina which reward schools for big performance gains without correcting for size end up rewarding small schools for random reasons. Worst yet, the focus on small schools may actually be counter-productive because large schools do have important advantages such as
being able to offer more advanced classes and better facilities.
All of this was laid out in 2002 in a wonderful paper I teach my students every year, Thomas Kane and Douglas Staiger’s The Promise and Pitfalls of Using Imprecise School Accountability Measures.
In recent years Bill Gates and the Gates Foundation have acknowledged that their earlier emphasis on small schools was misplaced. Perhaps not coincidentally the Foundation recently hired Thomas Kane to be deputy director of its education programs.
Ignoring variance and how it relates to group size is a simple but common error. As Wainer notes, building on a discussion in Gelman and Nolan, counties with low cancer rates tend to be rural counties in the south, mid-west and west. Is it the clean country air or some other factor peculiar to rural counties which accounts for this fact? Probably not. The countries with the highest cancer rates also tend to be rural counties in the south, mid-west and west! Once again, small size and random variation appear to be the main culprit.