LA Times Ranks Teachers

by on August 20, 2010 at 7:31 am in Data Source, Economics, Education | Permalink

The LA Times investigative report on teacher quality is groundbreaking. The teacher's union has already started a boycott but, as the shock recedes, I think this is going to be emulated throughout the country. It should have been done decades ago.  

The Times obtained seven years of math and English test scores from the Los Angeles Unified School District and used the information to estimate the effectiveness of L.A. teachers – something the district could do but has not.

The Times used a statistical approach known as value-added analysis, which rates teachers based on their students' progress on standardized tests from year to year. Each student's performance is compared with his or her own in past years, which largely controls for outside influences often blamed for academic failure: poverty, prior learning and other factors….

In coming months, The Times will publish a series of articles and a database analyzing individual teachers' effectiveness in the nation's second-largest school district – the first time, experts say, such information has been made public anywhere in the country.

Not much data is available yet but what is astounding is that the LA Times will release information on individual teachers. The graphic below, for example, is not an illustration it is real information on the real teachers named. To understand the importance of these differences note that:

After a single year with teachers who ranked in the top 10% in effectiveness, students scored an average of 17 percentile points higher in English and 25 points higher in math than students whose teachers ranked in the bottom 10%. Students often backslid significantly in the classrooms of ineffective teachers, and thousands of students in the study had two or more ineffective teachers in a row.

With better information there is a possibility that teachers will improve. Simply knowing that other teachers do better will encourage the lower performing teachers to ask why and to emulate best practices.

Unfortunately, we have little idea how to train good teachers. The best we may be able to do is to throw a bunch of people into the classroom and measure what happens but for that strategy to work it needs to be followed up with firings. Indeed, one recent study (see here for another explanation) found that the optimal system–given our current knowledge and the importance of teacher effects–is to hire a lot of teachers on probation and then fire 80% after two years, yes 80%.

I don't blame the unions for being up in arms and I feel for the teachers, for some of them this is going to be a shock and an embarrassment. We cannot simultaneously claim, however, that teachers are vitally important for the future of our children and also that their effectiveness should not be measured.  As systems like this become more common students will benefit enormously and so will teachers.

Moreover, I see this as a turning point. Once parents have this kind of information who will allow their child to be in a class with a teacher in the bottom ranks of effectiveness? And if LA can do it why not Chicago and Fairfax?

Many people said that information technology would revolutionize teaching but few had this in mind.

Addendum: Details on methods here.

1 Andrew Edwards August 20, 2010 at 7:47 am

for some of them this is going to be a shock and an embarrassment.

Thoughtful point. There is going to be a lot of gloating from those who see public education as disfunctional, and much of that gloating is justified.

However, it should bother us as well that John Smith may not have known that he was doing badly – maybe he’s trying real hard and what he’s trying just isn’t working.

Feedback really should not be provided in public like this and shame on everyone (unions, but also a board who did not bother to run the numbers) who let it get to the point where the only way for feedback to happen was in the LA freakin’ Times.

2 Jimbino August 20, 2010 at 8:01 am

Now if we could only get doctors and dentists to start publishing their prices!

3 Dave August 20, 2010 at 8:25 am

This study is also wholly dependent on the value of standardized tests. If thats how you measure teachers, then that’s what will be taught.

Only if “teaching to the test” is the most effective way to increase scores on the test.

4 Paul August 20, 2010 at 8:29 am

We have this in Massachusetts. And what happened? They changed the tests, constantly, so that no trend lines can be discerned.

I have great faith in the teachers union blowing through this like crap though a goose. Maybe faster.

Fughendabout it. It is the sleep of reformers reasons. It’s like Gorbachev giving motivational speeches to the Communist Party. Everyone knows the racket, except for the suckers…er…stiff.er…serf…peasants…citizens. Whatever.

5 Outsider August 20, 2010 at 8:36 am

I am deeply suspicous of these kinds of measures.
Metrics can and will be gamed.
Is it relative of absolute performance?
Also I don’t think educational goals are well defined nor do they
translate well to standardized testing.

6 Florida First Time Drivers Course August 20, 2010 at 8:42 am

Its really embarrassing to hear that LA is lack of good quality teachers.If teachers are not effective enough to teach students then how effective will be students……………..

7 mattmc August 20, 2010 at 8:48 am

Value-added is a vast improvement over the traditional measures, which don’t give a consideration to where the children started.

I am curious as to why they used ranked percentiles as the measure. It seems destined to look negative. Percentiles are zero-sum, and always seem to indicate failure. That is to say, as long as any group of students does better, one must always be shown to do worse. The reference to “backsliding” then would seem to be typical journalist lying with statistics BS.

It would be better to have a grade level assessment, where you say- these third-graders started at 2.7 but made it to 3.8 at the end of the year, making 1.1 years of progress. Meanwhile, the advanced kids in that class may have started at 3.2 but still only made it to 3.8, indicating that they were being held back by being in a class that was too slow…

The other factor that is ignored is the second derivative. If you have kids that can learn faster- they have the potential to show more progress. Slower kids have a lower potential for progress. This measure assumes that all classes learn at the same base rate. That’s why I’d like to see things like John Smith made .5 years of progress this year, instead of “moved from 55th to 45th percentile”.

8 Chris August 20, 2010 at 8:51 am

Percentile = relative performance. The tests taken in California do not include any measures of absolute performance; that is, they are not normed year to year. With some work they could be standardized, and then proper estimations could be run to account for things like measurement error. I’d call this an interesting good start.

9 Paul Nichols August 20, 2010 at 9:02 am

There are people who are working on methods to make teachers effective at teaching.

Take a look at this New York Times article from a few months back: Building a Better Teacher

10 DKB @ NYU August 20, 2010 at 9:13 am

I like the idea — some measurement has to be better than no measurement — but this has to be harder than the piece suggests. You have (at least) two issues: how to measure output (what have students learned) and how to correct for quality of input. The first is “answered” by the test. The second isn’t clear (year-to-year change?). But I hope this will instigate more work on how to do this better.

11 Buzzcut August 20, 2010 at 9:34 am

We don’t know how what makes a good teacher, or how to train teachers?

Then why on earth are there teacher colleges? Why do teacher licenses require that one graduate from such a program?

12 Pat August 20, 2010 at 9:44 am

If there’s any regression to the mean (meaning an individual student’s average performance over time), lots of teachers are going to want students who had a crappy teacher the year before.

13 floccina August 20, 2010 at 9:54 am

BTW to paraphrase and other commenter on another post:

All of this standardized testing and measurement is the inferior stuff that government does to replace allowing the customer to select the teacher that they want.

14 gulzar August 20, 2010 at 10:10 am

the full post is at

http://gulzar05.blogspot.com/2010/08/so-good-teacher-it-is.html

Here are four observations from the perspective of teacher value addition approach for developing countries like India

1. The critical challenge will be in the administration of the standardized tests. How do we manage the logistics of standardized examinations, given the wide geographical spread and massive numbers of students being tested? How do we ensure the purity of both the examination invigilation and paper valuation? In other words, how do we ensure the administration of the massive exercise of standardized tests without compromising on the purity of its results?

One way would be to outsource the process itself. However, its cost and more importantly, the perception and resultant salience of an externally outsourced assessment process will amplify opposition from the unions. Administering it through internal arrangements, for example by shuffling teachers across schools, too will raise formidable administrative and supervisory challenges. However, in the initial stages, this appears to stand the best chance of success.

2. A perception that such value-addition analysis would be used to assess teachers will naturally raise political opposition from the unions. It may therefore be necessary to completely de-link its use from high stakes decisions like punishing teachers.

In fact, mere disclosure of teacher-wise value-addition for each student and the entire class, will go a long way in contributing towards increasing performance outcomes. Appropriately designed student report-cards aimed at parents, teacher-report cards aimed at administrators, and school-report cards intended for community at large, can play an important role in getting all stakeholders to respond in a manner that will nudge teachers to improving their performance.

3. In order to buy acceptance among teachers, such value-addition analysis should be spun-off as say, “teacher enhancement feedback programs”. Analysis of classroom data and student learning outcomes can be used to deduce specific skill-deficiencies of teachers. This can in turn be used to objectively design training programs and impart focused trainings to teachers based on their respective deficiencies.

4. The biggest source of last-mile challenge will be in ensuring that the data collected, analyzed and presented is acted upon. It is commonplace in government to have massive data being collected and not being utilized in any meaningful manner. And the sheer volume of longitudinal data collected only increases the probability of policy-making getting buried in the small detail of numbers.

As aforementioned, this last-mile problem can be overcome with effectively designed and institutionalized policies that uses the data to simultaneously inform parents about their students’ performance, administrators about the respective value-addition (and value-subtraction) of teachers and performance of schools, and teachers about where they and their students are lagging behind.

This information disseminated in the most cognitively effective manner (well designed report cards), through platforms like school management committees, and utilized to design training programs for teachers and remedial classes for students, can go a long way towards improving the quality of our education system.

15 bork August 20, 2010 at 10:25 am

One of the interesting characteristics of this study is how cleanly you can study a teacher as a ‘treatment condition’ in the great public-school experiment.

What this says to me is that John and Miguel need to spend more time in each other’s classroom. As it currently stands, there is next to no learning or diffusion of practices among teachers.

If teaching ability is somehow a pure individual-difference variable, then sure, hire indiscriminately and fire the vast majority.

But if teaching is not some mystical art, if it is an occupation like other occupations, then we should expect social learning, feedback, social comparison, information sharing, and a host of other well-known group processes to kick in.

Talk to a teacher you know and ask how many times they’ve had a colleague sit in on one of their classes for friendly informal feedback. I think the lion’s share of teachers are observed teaching only during rare performance evaluations and other formal observations. No wonder teachers’ skills, ability and talent don’t spread to their colleagues like you would expect in a regular work team.

16 Tim August 20, 2010 at 10:29 am

I doubt this will be a bad thing for public education. If you do the same thing with private schools, you’ll quickly find that parents are paying more for less. Private schools almost always have poor teachers, but students who are self-selected for success. Thus appearing to be a “good school”.

I worry about the effectiveness of this, though. The idea of a “good school” is still primarily about what percentage of the students are white and not getting free lunches. See this article:

http://www.familycircle.com/family-fun/money/10-best-towns-for-families-2010/

So will this lead to better informed parents or will we still judge the qualities of schools based on the same criteria they always have? I lean towards the latter.

The one positive is that the good teachers will probably get the respect they deserve.

17 dale August 20, 2010 at 10:32 am

I also have concerns about standardized tests as well as regression to the mean (or the reverse – that there is built in momentum for students to rise or fall, independent of the teacher) – but, there is still potential merit to the approach. I’m willing to consider it further and would love to see the data.

One question that occurs to me and hasn’t been mentioned yet, is that getting this data means that students are tested more than once each year. My own son is tested yearly and I think far too much attention is devoted to the standardized tests – potentially at the expense of what I think the schools should really be doing. Does this mean that Los Angeles tests their students twice yearly? Is this a common practice? It is very hard for me to see the justification for testing students at the beginning and end of each year. It will be a miracle if students are left with any enthusiasm, creativity, or brain cells worth bringing to their next year’s classes.

18 blink August 20, 2010 at 10:50 am

Regression to the mean could be important here, especially to the extent that test scores vary (measurement error, luck, etc.). Perhaps a way to “look good” is to teach kids who score poorly, some of whom have underperformed previously; a way to “look bad” would be to teach only kids whose earlier test scores were high, including some overachievers.

19 casey August 20, 2010 at 10:55 am

You can talk all you want about it not being a perfect measure.. but I want my kid in Miguel’s class… and I bet you do to…

20 Brian August 20, 2010 at 11:13 am

one thing i have not seen regarding the “value-added” approach, that i wish Tyler would comment on, is the effect of diminishing returns. Although other factors like socioeconomic background and such are already controlled for in the models, isn’t it possible that it would be easier (or more difficult) to have students show a 10% improvement depending on how much they are already achieving? for example, for students who perform poorly, it might be easier to get a 10% improvement out of them. For students who are already achieving at the top, it might be harder to get a 10% improvement.

an analogy is someone’s time running the 100 meter dash. if you can only run it in 20 seconds, a 10% improvement takes you to 18 seconds. But asking Mr. Bolt to improve 10% would take his 9.7ish down to the 7 second range, a near impossible task.

but from the brief description of this value-added approach, this is not one of the factors that is controlled for. what am i missing? They said that economists developed this technique. Tyler, can you comment on this? thanks.

21 Nate August 20, 2010 at 11:21 am

Gabe Harris (tangentially) makes another good point that occurred to me on the Metro this morning. Don’t a lot of you here claim to be libertarians? Yet you want standardized, government-mandated metrics to be the basis upon which we judge teacher performance? Right. Sometimes I could swear a lot of conservatives/libertarians who come out in favor of standardized testing are only doing so because they believe it will undermine the public school system itself in the long run.

22 vasantha August 20, 2010 at 11:23 am

Pushy parents can also send their kids to cram schools to prepare for the tests, so that the kids can get into gifted students’ program. In Irvine, CA, Dr. Steven Choi, runs one such outfit, was a school board member, later became city councilman and is now running for state assembly.

23 mattmc August 20, 2010 at 11:30 am

I find all of the complaints about “standardized” tests laughable, as if tests are bad if they are standardized. Are unstandardized tests somehow better, do the standardized tests need to measure something different, or do you all think that it is impossible and irrelevant to measure whether students can add, read, and know if WWII came before or after the Vietnam War?

24 sm August 20, 2010 at 11:34 am

Endogenous assignment of students (send weak students to Mr. Aguilar, he’s great with them!) + regression to the mean could account for these results.. plus it is very suspicious there aren’t confidence intervals around that figure; there probably is no statistically significant difference between those slopes.

25 Lou August 20, 2010 at 11:45 am

How do you measure the success of a business manager- Stock price? Sales growth? Market share? Margins? Customer reviews? I’m sure there is disagreement. And yet I have never heard anyone suggest it shouldn’t be measured at all.

26 Morgan Warstler August 20, 2010 at 12:09 pm

The next step is to allow the better teachers to choose to take on larger class sizes and get paid more for doing so.

27 Steve August 20, 2010 at 12:34 pm

As for regression to the mean, why would you want to teach the kids that did worse? Wouldn’t you want to look at a kid’s “normal” percentile rank and get the kids who did worse than normal FOR THEM last year? e.g, You’d rather have the 95th percentile kid who got 90th last year than the 40th percentile kids who got 50th last year.

28 dwr August 20, 2010 at 12:57 pm

According to many studies, the SAT is very g-loaded… er, or at least it was before the writing section was implemented; I don’t know of any more recent studies concerning the SAT and g.

I guess my point is that all this “teach to the test” rhetoric kind of ignores the reality of what standardized tests measure, and perhaps even the point of standardized testing in the first place.

Anyway, regression toward the mean and diminishing returns can all be accounted for with clever statistical methods. It’s an issue, but it’s an issue that the LA Times can control for if they’re smart. And with large enough sample sizes and a good degree of stochasity (i.e. students are randomly assigned teachers), then it all averages out in the end, anyway, and perhaps it isn’t that big of an issue.

29 Anonymous August 20, 2010 at 1:34 pm

I’m not at all sympathetic to the teachers who object to being evaluated. They grade our children as if a student’s success or failure was entirely the student’s doing. I have no objection at all if they want to dispute the methodology, so long as they provide better measures that we can use.

30 hotdrop August 20, 2010 at 1:51 pm

This is a great thing for education because it will cause a stack the deck effect. As soon as these scores get published parents that are heavily invested in their students education will push school administrators to put their kids with the best teachers. This will move kinds with the best chances of success towards the best teachers causing their scores to skyrocket forcing more good students their direction. The unfortunate consequence being of course that students that have poor family support in their education will get left in the dust. They will continually get pushed to the lower grade teaches and have more difficulty succeeding. Ive seen quite a few studies that show that there is a strong correlation between parent involvement and success in education so overall better success for students whose parents are involved may not be a horrible allocation of resources.

31 KevinH August 20, 2010 at 2:28 pm

I think this is certainly a great first step in quantifying teacher quality, however it is true that percentile rankings do have a lot of quirks. Mostly, percentile rankings can tell you, in a group, who are the best teachers and who are the worst, but you need to be careful how you define your group. If the percentile rankings are a year to year measurement they will much less useful than if they use 5 or 10 year running averages. Even then we run into a problem of if we have 10 years of great teachers, the only-kind-of-great teachers start to look very bad. But, at the very least, this should start separating good from bad teachers, and perhaps other non-relative measures can be found.

Also, for those concerned about seed and regression to the mean, you can take a look at the percentiles themselves. John Smith got a seed of smarter students, yet neither teacher started above the 50th, so both should be expected to go up. That takes out explanations of both ‘the smart getting smarter’ and regression to the mean as the difference between them.

32 anon August 20, 2010 at 2:57 pm

This will likely confirm the information about teachers that parents have been sharing for years.

I agree with the commenters above that the public school teachers union will likely figure out ways to make future comparisons impossible and thus insure that all teachers are treated “fairly”. Forget about the students.

However, the teachers unions can not put off forever measuring teacher effectiveness when teachers are well paid and have good pensions as public finances become more strained. The public is waking up.

It never ceases to amaze me that many of the same folks who are upset with corporate greed, evil corporations and for profit businesses, seem to believe that teachers, government workers and “nonprofit” employees, can do no wrong – a strange kind of religious faith.

33 Chris T August 20, 2010 at 4:15 pm

While they shouldn’t be the be all and end all of schooling, what do the people against standardized testing proposed in their stead for objectively measuring student progress?

34 Doc Merlin August 20, 2010 at 5:12 pm

@mattmc

Where did you do your schooling, that is a brilliant idea.

35 Joe August 20, 2010 at 7:31 pm

How many ways is there to teach 2+2=4? how many different ways are there to spell a word? A standard education implies that all students should be taught the same things in the same way. This requires teachers to teach the same things in the same way.

The children are getting cheated out of a good education, because unions do not want to allow standards to apply to teachers. This would expose teachers to scrutiny based on whether the children are meeting up to the standard. If children are not meeting the standard, then the teacher’s performance would be questioned.

36 Matt August 21, 2010 at 3:58 am

Joe,

Exactly. There’s really only one way to do heart surgery, design a bridge or write a patent. Sure, there’s some leeway given, but successful businesses standardize their processes and rigorously control for anomalies. Processes can be changed or improved, but some sort of norms and standards are required.

Teaching has apparently been the special exception to this rule. Or I should say teaching in America. Other countries have much more formalized methods of education to basically drill students in math, history, vocab, etc. And guess what? They do a whole lot better than we do.

Teaching isn’t rocket science. A good teacher, like a good manager, pushes kids farther than they thought they would go and supports them actively along the way. But that’s too much to ask from a lot of teachers who would rather coast by and collect their 80% pensions.

37 T Bond August 21, 2010 at 7:43 pm

This guy is right up there with the old timey “efficiency experts.† Humanity gone, unable to look at the situation from any other direction than his own mechanical fixation. All students, like all teachers, are identical machines. No difference in cultural background, family situation, interests, goals or lack of the same, one best teaching method covers them all.

There is an infinite number of people out there who want to teach, so you can treat them like recruits in war time. Rigid stratification is the way to go. The children, like the teachers, love to jump through a hoop. Ubermensch can always define a system to improve democracy.

38 Troy Camplin August 22, 2010 at 5:11 am

That’s the only kind of standardized testing that could work

39 Smithy August 22, 2010 at 9:12 pm

teachers/schools can spend a lot of time raising their standardized test scores by focusing almost all of their classroom time on teaching those tests. Is this what we want to teach our children or should we teach them to think, analyze and create? I don’t want people who can memorize a bunch of facts but do not actually understand the problem working for me.

40 Nick August 23, 2010 at 9:32 am

If they are going to name teachers, then they should name students as well. Or are students not ultimately responsible for their own performance?

41 mcse August 25, 2010 at 8:15 am

The teacher should have proper knowledge whatever he teach but nice post

MCSE

42 air max September 15, 2010 at 9:25 pm

Hey friend, if i say i buy one cheap air max, air max 90, nfl jerseys authentic fron the website only $41.11,do you want to believe?

it is really a good website, they say all their products are authentic, i like the products ghd purple,

and christian louboutin, at first i dont believe that, just have a try,

and now, when i receive the products i am so happy, for their good quality and best service.

i take my air max 95 and nike shox just only 3 days for shipping. and it is really a nice style.

thanks so much.

43 aewga November 18, 2010 at 3:49 am

The Times used herveleger a statistical approach known as value-added analysis, which rates teachers based on their students’ progress on standardized tests from year to year. Herve Leger Strapless Each student’s performance is Herve Leger Dresses compared with his or her own in past years, Herve Leger Skirts which largely controls for outside influences often blamed for academic failure: poverty, prior learning and other factors….

44 True Religion December 24, 2010 at 9:13 pm

We sincerely hope that all the customers from abroad and nation can share the benefit with us.

Comments on this entry are closed.

Previous post:

Next post: