Getting beneath the Veil of Effective Schools: Evidence from New York City

That is a new paper by Will Dobbie and Roland G. Fryer Jr., here is the abstract:

In this paper, we collect data on the inner-workings of 39 charter schools and correlate these data with school effectiveness. We find that traditionally collected input measures—class size, per-pupil expenditure, teacher certification, and teacher training—are not correlated with school effectiveness. In stark contrast, we show that an index of five policies suggested by qualitative research—frequent teacher feedback, the use of data to guide instruction, high-dosage tutoring, increased instructional time, and high expectations— explains approximately 45 percent of the variation in school effectiveness. The same index provides similar results in a separate sample of charter schools.

The gated AEJ version is here, an ungated version is here.

Comments

Interesting about the correlations, because some of the variables should also be correlated with each other, but apparently are not.

So, no effect of class size, but an effect for frequent teacher feedback and high dosage tutoring and increased instructional time. Hmmm. How do you do that with large classes. And, if you also extend the class day to achieve those results, or increase class time, you would expect to see increased per pupil expenditure.

Truly amazing. Water into wine.

Marginally reducing class size is a ham-fisted way to achieve individual attention and remediation while tutoring is more direct, for example. Aside from that, one might assume the non-correlations are arrived at "after correcting for" the correlations.

Aside from that, one might assume the non-correlations are arrived at “after correcting for” the correlations

Not if we're trying to suppress the results.

Those correlations you are assuming cannot be simply assumed, but must be proven.

The amount of teacher feedback and tutoring is obviously not solely determined by class size. There are also issues like teaching style, amount of teacher effort, and the amount of support given by staff other than the primary teacher. Yes, it does seem logical that larger classes would limit a teacher's ability to do give one-on-one attention, but this is just one factor among many. Thus it is quite possible that there is no correlation.

With expenditure per pupil, it obviously depends on what they are spending the money on and how efficiently they are using it. (I haven't heard anyone arguing "The U.S. spends much more on healthcare, so obviously we must be healthier" lately).

Gee, Dan, you really are strict when you say "teacher feedback and tutoring is obviously not solely determined by class size" I don't live in a world where something is "solely" determined by something else. Maybe you do. The other variables you list--"teaching style, amount of teacher effort, and the amount of support given by staff other than the primary teacher."--were not variables in the study (except perhaps teacher training if it correlates with good teacher style), so, as you say, "Those correlations you are assuming cannot be simply assumed, but must be proven."

Sorry if my original comment was not clear.

You seemed to be assuming that there MUST be a correlation between class size and the amount of tutoring and teacher feedback received. Therefore there must have been something fishy about the study, because it showed the latter to be important but not the former.

My point was that there is no reason to assume a correlation between the two. Yes, it seems there would be an intuitive relationship, but there are also many factors that could possibly influence something like the amount of tutoring received. I didn't intend to claim that these other factors ARE the reason, just to point out that there are many possible factors, and it is not clear which are most important. Finding no correlation between class size and tutoring is an entirely plausible result.

but it is an obvious correlation.
look at a class with 100 students.
look at a class with 2 students.

dan111, You misrepresented what I said. What I said was: "So, no effect of class size, but an effect for frequent teacher feedback and high dosage tutoring and increased instructional time.", whereas you represented that I said "amount of tutoring and teacher feedback". First, I included increased instructional time. Second, it was not increased tutoring, but high dosage tutoring that the report covered. Both of these must have some relationship to increased spending relative to those who do not provide these three elements. Why isn't there a correlation?

Bill,

There could be enough spending on ineffective alternative items that the correlation between per pupil spending and effective tutoring is very weak. For example suppose the base is $10,000 per pupil. 10% of group that spends $11,000 per pupil uses extra money for high dosage tutoring on the weakest subject. The other 90% of the group that spends $11,000 per pupil, spends the extra money reducing class sizes, anti drug programs, museum field trips, extra overhead, etc If 90% of the extra spending is spent ineffectively there will be a very weak correlation.

Sortof, True, but I wonder if something else is going on re spending. I checked out some of the authors other work, and what is interesting is that, remember, this is study of charter schools, which means spending for teacher salaries varies over the number of schools, which means that some schools spend less, others more, because of different teacher salaries. (Try to find that variation in unionized public school system.). Second, the number of class periods and the total time devoted to teaching in a charter environment may vary more than in a public school environment (ie, less admin time, leave at 5pm rather than 4 pm). When you take the mix of different spending levels (due to salary variation) and different amount of time spent teaching, even between charters, it might be the case that what you are seeing is more time spent teaching, at a lower cost. OK, so, what does that mean for a public system: it would mean that you would have to pay teachers more to get them to stay later, teach more classes, etc. to have the same results.

Google the author's name and look at some of the work. If you are prepared to pay more for more teacher effort in a public system, it might be worth it.

The tutoring they refer to is generally intervention style. Most leading charter schools have assertive intervention policies to keep kids from falling behind. (As opposed to tossing folks into a "slow" classroom that is taught to lower expectations.)

The reason that dollars don't correlate is because class size doesn't correlate, and teacher salaries are the largest component of school costs. (The building is usually less than 20%, and there are many more teachers than admin staff)

We parents raise money to hire in-room adjuncts, which increases high dosage tutoring but not budgeted cost. (Although, bless their hearts, the union tried to kick the adjuncts out unless they became an unaffordable line item; currently a hot topic in the board elections).

And reducing teacher admin loads can increase class time since the teachers call half-days whenever they attend training or conferences. Field trips, too.

They can be correlated, but if you are including multiple instrumental variables, the better ones (which more directly estimate the target) are going to be determinative in the regression.

That is, class size is at best a proxy for direct teaching time. Once direct teaching time is measured, the proxy should fade in significance.

Simply put, a big class with the teacher actively engaged in proper teaching method is going to show better than a small class where the teacher is not using effective method (say, he sits at his desk while everyone fills in worksheets).

Who'd have thunk it... the most effective measure of teaching effectiveness is actually... the amount of teaching done.

"class size, per-pupil expenditure, teacher certification, and teacher training": anyone whose instincts told him that these would matter greatly must be blind to human nature.

'“class size, [excluded], teacher certification, and teacher training”: anyone whose instincts told him that these would matter greatly' would probably resemble someone like Profs. Cowen or Tabarrok, actually. Not to mention pretty much the rest of GMU's faculty, especially those teaching graduate or doctoral courses.

Or they would resemble the Finns that created and run the world's best educational system, if one accepts the idea of comparative testing over a longer time series to be worth evaluating. And the Finns even believe per-pupil expenditure is important, though not in a limited classroom sense, as the Finns go to considerable lengths to expend money on potential students, starting before birth.

So you are attributing the Finns' unique success to their uniquely large welfare state? I question that.

American schools have better results than Finland when you adjust for genetics

Pray do elaborate.

The honor goes to Tino Sanandaji
http://super-economy.blogspot.com/2010/12/amazing-truth-about-pisa-scores-usa.html

That very link shows that Finland dominates the demographically adjusted PISA scores, putting the lie to mike's statement

Finland is a small homogenous country in which the prevalent cultural trait is perseverance. 'Susu' - loosely translated as guts - could almost be a national word.

Comparisons with a large and culturally diverse country like the US are absurd.

As for class size my kids had 25 children in their classes in Canada along with a teaching assistant. Homework was nil to light and the amount of time dedicated to academics minimal. In Singapore they have 40 kids in the class, lots of homework, the teacher available online after school and a very steep learning curve.

You mean "Sisu", right? I would be surprised if the Finns are more dedicated to school than the Chinese or Koreans, but I guess I can't be sure. They seem to have a fairly different school setup than those countries, though. Has Finland always outperformed?

Am I missing something? Is there value in explaining only 45% of the variance? What other possible variables could explain improvement?

Well, the researchers focused on "five policies suggested by qualitative research". It could be that schools focusing on those policies tend to have administrators that follow and care about educational research because they are more competent than administrators elsewhere - that could result in better hiring practices and more effective teachers at the schools they run. Probably a bunch of effects like that; as mentioned below explaining half of the variance is nothing to dismiss.

@Richard Russell
In social sciences, explaining 45% of the variance is a big amount. And the other important thing is that the remaining 55% of variance is not explained by traditional inputs that are the focus of many education policies.

You would think by now that everyone, even social science researchers, would have figured this out. Good students from good homes results in good schools. Fill up the school with featureless bureaucrats from the local teacher's union and parents will force a change to better teachers who provide feedback, use data and provide high dosage tutoring. Build an ideal school in the ghetto and the only way it works is if you can skim off the smart fraction from homes with parents looking for a way out of the ghetto for their kids.

Their control group is made up of students who applied for admission to the same charter schools but were rejected by random lottery. Which makes your point irrelevant to this paper.

Just keep those fingers in your ears and the bad man saying bad things will go away.

I was just responding to your claims of selection bias ("skim off the smart fraction...") by pointing out that the study used a very robust method of controlling for this problem.

Fair enough. I was not critiquing the specifics of this study, just pointing out the herd of elephants in the room.

You're making an ass out of yourself

If it is all about the kids and their parents, why spend so much money?

If only the right kids showed up. If only so many people didn't log on to follow the law that we wrote. The line in Canadian health care for years was if only so many people didn't get sick at the same time.

There's not even a correlation between money and results, much less evidence of causation. The fact is the schools themselves and the massive parasitic bureaucracy that feed off of them are irrelevant. But, we will keep throwing billions at them and the social scientists who churn out reports supporting the latest fads. At least will will until the money runs out.

In a general sense you are right, but come on. Schools do matter at the margins. What is your theory on Finland?

Small, homogeneous, good genes

Finland is Finland because it is full of Fins. They have their culture, language and history, which is unique to them. How they arrange their schools is not what makes them Fins. They arrange their schools as they do because they are Fins. Swap the population of the Central African Republic with that of Finland and you are not getting an economic basket case, incapable of capitalizing on its vast natural resources.

That doesn't explain why Finland dominates the demographics-adjusted PISA scores versus the other Scandinavian countries and other homogenous countries with good genetics such as Japan.

If its small, homogeneous and good genes that explains Finland, how to explain the worse outcomes for Sweden and Norway?

One difference may be that Finnish students are expected to learn at least two other languages (English and German, I believe, possibly more), and this introduces greater rigor, earlier, to the educational experience. Maybe the Finns get more out of their students because more is expected.

@Marcus

Sweden has higher nominal GDP, so do differences of those magnitudes in PISA show anything important? I think not. Does PISA measure anything important at all? Maybe not.

Obviously, smart, conscientious kids from good families are the biggest factor in successful education, but if you can get marginal improvements in teaching methods, while possibly even spending less money, why not try? Isn't it worth it if 11 out of a 100 instead of 10 out of a hundred manage to pull themselves out of the ghetto?

Ah, the old "if we can save just one life" argument. What if that one extra kid out of the ghetto turns out to be Ted Kaczynski? We can't know these things. What we do know is beyond the basics, the school plays a trivial role in education. Genes and the parents play the decisive roles.

This may be one of the best pieces on charter schools, but how does this paper get published/recognized without so much a note or reference about the meta-analysis by Hattie? Hattie reviewed 800 meta-analysis that included 80,000 studies and god knows how many students within them. http://visible-learning.org/hattie-ranking-influences-effect-sizes-learning-achievement/ I suppose you can never have too many confirmations in social science but still.

"frequent teacher feedback, the use of data to guide instruction, high-dosage tutoring, increased instructional time, and high expectations"

These things are tutoring. I have not read the paper, but if the abstract is representative of the paper the conclusion is that tutoring was effective.

In effect, we've reduced classroom size down to one.

My wife and I both tutor, and we are more effective than the public school teachers. This is expected since we work one on one and actually know the subject we are teaching. Neither of us could teach a classroom and get the same results.

Isn't all this dead obvious? Is there something new here I'm missing?

I support the idea of using technology to provide the classroom level instruction with human interaction doing the tutoring.

This result is far more important than you give it credit for. Lots of tutoring is obviously expensive. But in this case, per-pupil expenditure is not impacting the outcome. So apparently spending more time tutoring and less time on other stuff is a good strategy. That's something that can be used to get better results without higher cost.

In the robot future tutoring may be one of the jobs people can actually get, classic service work, hard to fully automate.
And since it won't pay much unless you are tutoring the rich, it won't help the inequality issue...

My neighborhood is full of tutoring centers. Even kindergarten kids go for tutoring - it's nuts. The pay for the employees is terrible - they use HS kids to do the tutoring. When I was a younger you got tutored by an adult who came to your house - but in today's terrible economy it's cheaper to send your kids to a McDonald's style tutoring center where they can get tutored by a HS kid.

This may be the future of schooling though, since service jobs are all that will exist, every kid will be tutored perhaps in groups of 3-4. Teachers will make even less, but it's a job. Kids might actually learn more though.

One teacher tutoring 30 kids, one at a time, while the other 29 work on self-directed work (or even just go play) is more effective than one teacher lecturing 30 kids for the same amount of time.

It's not the class size that matters, it's that individualized instruction is much more effective than lecture.

My local public school district spends 990 hours over 6 years per student "teaching" each student to read and still fails almost half the time. A competent local tutor accomplishes a much better result in only 30 contact hours.

This is why the Montessori Method is so effective and why schools that actually take into account what a student needs to learn in order to make progress are so much more effective than the "standard" public school lecture methodology.

I'm mostly curious about what sorts of things are implied by "use of data to guide instruction".

Maybe similar to the police techniques that changed NY. Crime is high somewhere? Put officers there, deal with it. Oddly it seemed to have worked. There was a notion that nothing could be done about crime. There seems to be a notion that nothing can be done about kids not learning.

Data would tell you that this child doesn't quite get reading, or some level of arithmetic. So put a bit of effort into assisting the child to learn what it needs. It doesn't have to be complicated; even a weekly or monthly sit down going over how each child is doing and deciding a course of action would probably work. The further away the data from the subject requires more complex data collection, but close in, not so much. Then writing it down and seeing if it changes, if not, respond. If the teacher doesn't do it, find another one who will.

I'm certain good teachers do this all the time. And good administrators would set up something formal or informal to do it. Making sure the rest of them do it will maybe make them into good teachers. I suspect the raised expectations and scrutiny that these charter schools undergo helps them keep focused on what they are being paid to do.

+1 to derek

This comparison is far from perfect. The street criminals in NY (and elsewhere in the US) were "dealt with" in the sense that now they live in the largest prison complex on the planet and victimize one another. It may well be that the total level of crime stayed the same, but much of it now takes place behind the bars (which does not interest the voters at least).

An analogy of such "dealt with" in education: you would create "gutter" schools where 3-4 per cent of the worst pupils would be forced into, and wouldn't count their outcomes towards the national average. The median voter would be probably almost as happy with such solution as he is with the prison situation above.

My +1 was not about the policing analogy, it was for the last 2 paragraphs.

does 'frequent teacher feedback' mean - Principal and other teachers giving teachers feedback.. or Teachers giving students feedback.

Maybe a bit off target, but for those interested in learning more about NYC charter schools I recommend this inside look by John Owens:

http://www.amazon.com/Confessions-Bad-Teacher-Shocking-Education/dp/1402281005

Owens bailed on his publishing career to become a NYC teaching fellow. His story details how challenging it is to work as a teacher in a NYC charter school. There's information on the push to make everything data driven (e.g. enter everything you can into spreadsheets) and homogenized. One thing not often mentioned in education debates is management's impact on the teachers. There are some truly sadistic principals out there. Also not mentioned in this morning's link is the word "cheat" - as many of you know, there are massive principal-teacher driven cheating scandals involving schools. So one needs to be careful when looking at standardized test scores across schools. Heck, I remember 25 years ago one of my teachers walked the room during a standardized test and told me to review one of my answers. Back then I thought it was really nice of her. Now I know why she did it, LOL.

Bottom line - I was thinking about a career change to teaching. But Owens' book plus conversations with a DC teaching fellow changed my mind. The school day is 730AM to 5PM (which doesn't include lesson plans, test grading, satisfying the data beast, report cards, parent outreach, etc.), you're attacked by the kids, teachers, management and press and the pay stinks.

I come from a family of teachers and always assumed I would become a teacher. But the bloom is off the rose - it's a demanding, thankless profession. For those that think otherwise, check out Owens' book.

Are the spammers getting more sophisticated?

I'm not a spammer! I just read the book and thought it was great. There are a lot of anti-teacher folks on these boards and this book was illuminating. These charter schools aren't much better than the public schools and I believe the data bears that out.

If you don't want to pay for the book you can read an article length version of it here:

http://www.salon.com/2011/08/29/confessions_of_a_bad_teacher/

Well, you do get a summer vacation. Are private schools better than public? I would think the bureaucracy would be a big source of this problem.

The summer vacation is valuable but many teachers use it to earn extra income by working in a camp. My buddy was a DC teaching fellow - he had to make home visits during the summer. He had to visit each student and their family - he introduced himself and talked about the work they were going to do the following school year. It's "feel good" and makes the charter schools look good but it's totally ZMP.

Private schools have fewer issues with unruly kids but the salaries are MUCH lower. Teachers have to choose between rougher kids and more money vs. better kids and less money. Of course when the kids are very affluent that brings a different set of problems (e.g. helicopter parents, entitlement, etc.).

The summer vacation is valuable but many teachers use it to earn extra income

but?

For better commentary on what makes a school great, read Amanda Ripley's The Smartest Kids in the World: And How They Got That Way. You'll understand why American teacher's use of feedback is excessive. You'll also learn about the $4 million teacher -who is really just a cram specialist - and why he thinks Finland is a better model for schools than Korea. Yes, Finland, where they have strong teacher's unions! Koreans actually are trying to get rid of their system. Also, technology in the class room is very overrated. Great book.

I wonder why people think that "teacher's unions" are the same thing across cultural boundaries. They are not. Have you ever served in the military? Do you think that American military is basically the same as Russian military?

Each country has a very specific social compact and norms regarding interactions of unions and employers. The American model of unionization isn't in any sense universal and I would say that, globally, it is one of the less successful ones (tight links between unions and a major political party seem to be problematic everywhere in the world). You cannot transplant the Finnish experience to the American one, the cultures are too different.

Maybe more pay for less work plus barriers to entry correlates strongly with something to do with teachers rather than something to do with students. Dunno.

Comments for this post are closed