Teacher Performance Pay: Experimental Evidence from India

by on September 30, 2009 at 7:39 am in Economics, Education | Permalink

In an impressive new paper, Karthik Muralidharan and Venkatesh Sundararaman provide evidence on the power of teacher incentives to increase learning.  The paper is impressive for three reasons:

1) Evidence comes from a very large sample, 500 schools covering approximately 55,000 students, and treatment regimes and controls are randomly assigned to schools in a careful, stratified design. 

2) An individual-incentive plan and a group-incentive plan are compared to a control group and to two types of unconditional extra-spending treatments (a block grant and hiring an extra teacher).  Thus the authors can test not only whether an incentive plan works relative to no plan but also whether an incentive plan works relative to spending a similar amount of money on "improving schools."

3)  The authors understand incentive design and they test for whether their incentive plan reduces learning on non-performance pay margins.

The results are as follows:

We find that the teacher performance pay program was highly effective in improving student
learning. At the end of two years of the program, students in incentive schools performed
significantly better than those in comparison schools by 0.28 and 0.16 standard deviations (SD)
in math and language tests respectively….

We find no evidence of any adverse consequences as a result of the incentive programs.
Incentive schools do significantly better on both mechanical components of the test (designed to
reflect rote learning) and conceptual components of the test (designed to capture deeper
understanding of the material),suggesting that the gains in test scores represent an actual
increase in learning outcomes. Students in incentive schools do significantly better not only in
math and language (for which there were incentives), but also in science and social studies (for
which there were no incentives), suggesting positive spillover effects….

School-level group incentives and teacher-level individual incentives perform equally well in
the first year of the program, but the individual incentive schools significantly outperformed the
group incentive schools in the second year….

We find that performance-based bonus payments to teachers were a significantly more cost
effective way of increasing student test scores compared to spending a similar amount of money
unconditionally on additional schooling inputs.

Surprisingly, since absent teachers are a big problem in India, reduced teacher absenteeism per se does not appear to be the primary mechanism by which incentives improve learning.  Instead the primary mechanism appears to be more intensive teaching, including more homework and classwork and better attention to weaker students, this greatly increases the relevance of these results to teaching in the developed world.

Addendum: See also Karthik's comments on the comments at 26.

1 Jameson Burt September 30, 2009 at 8:47 am

I am rectified
I argued just last week against workplace incentives,
using mental gymnastics, not statistics.
But definitive answers spout from the mouth of statistics. Left with only the minuscule doubt that we must reserve for any view on any subject, I will now talk and act as if teacher incentives are efficacious.

My and others mental gymnastics against teacher incentives was/is easy and unbridled. Statistical studies, and this large study in particular, are enormously difficult particularly in time. Muralidharan and Sundararaman paid the price to exact the truth.

While people might call for new affirmative studies, to outright argue against teacher performance pay should lead us not to question these statistical results but to severely question the intransigent in an impolite way.

I wonder how much of my thinking amounts to slosh, when a study like this forces my belief into a U-turn.

2 Jeffrey Horn September 30, 2009 at 10:00 am

Well, no one can accuse Jameson Burt of confirmation bias. That has to be the most lucid and enjoyable narrative of Bayesian updating I’ve ever heard.

3 David September 30, 2009 at 10:45 am

Just as workers are less productive on an incentive program over time, I wonder if teachers would be as well. Sure the scores increased for the students, but 1. Who’s to say the scores truly showed valid progress; 2. Will the heightened performance continue beyond 2 years?

A teacher motivated by incentives receives their motivation externally. Motivation in such a form is never sustainable. Over time, I would expect the motivation teachers receive from incentives would decrease over time.

I agree, this is an important study and it is great that incentives do truly show increased performance. Great. However, I truly believe there are much better solutions.

P.S.
Jameson, you are a master at saying absolutely nothing. Good job.

4 tim September 30, 2009 at 11:25 am

This is not in any way comparable to anything in the United States, because the ability and desire of parents to help their children learn is very different in India compared to the United States. Put simply, Indians respect education a great deal more, and parents strive to help their children achieve.

In the United States, there is a significant correlation between student performance and parental involvement; indeed, the influence of the teacher on student performance is extremely minor.

5 Dbltap September 30, 2009 at 11:56 am

Its civil servants are not known for their intrinsic motivation.

Intrinsic motivation is the first thing that pops to mind when thinking about American public servants.

6 Pockets September 30, 2009 at 11:58 am

A fantastic paper. Resistance to performance pay for teachers here in the UK remains extremely vocal, despite the fact that we supposedly already have a (very lax) performance pay system. Sound experimental evidence can only help the case for a more sensible remuneration system for teachers (he says optimistically…)

7 Anders September 30, 2009 at 12:24 pm

@JSK:

And where exactly are civil servants noted for their intrinsic motivation?

8 eh.nonymous September 30, 2009 at 12:33 pm

what’s the difference between the word effective and efficacious? i didnt know statistical studies were enormously difficult in particular to time. i appreciate you shedding light on that.

so if india and the us have different levels of intrinsic motivation, how would you:

1)determine that exactly
2)control for that in a model

9 Arun September 30, 2009 at 1:23 pm

Read the paper:

The average rural primary school is quite small, with total enrollment of around 80 to 100
students and an average of 3 teachers across grades one through five.

One teacher typically teaches all subjects for a given grade (and often teaches more than one grade simultaneously). All regular teachers are employed by the state, and their salary is mostly determined by
experience and rank, with minor adjustments based on assignment location, but no component based on any measure of performance. The average salary of regular teachers is over Rs. 8,000/month and total compensation including benefits is close to Rs. 10,000/month (per capita income in AP is around Rs. 2,000/month; 1 US Dollar ≈ 48 Indian Rupees (Rs.)). Teacher
unions are strong and disciplinary action for non-performance is rare.

….

Regular civil-service teachers in AP are transferred once every three years on average.
While this could potentially bias our results if more teachers chose to stay in or tried to transfer
into the incentive schools, it is unlikely that this was the case since the treatments were
announced in August ’05, while the transfer process typically starts earlier in the year. There
was no statistically significant difference between any of the treatment groups in the extent of
teacher turnover or attrition, and the transfer rate was close to 33%, which is consistent with the
rotation of teachers once every 3 years (Table 1 – Panel B, rows 11-12). A more worrying
possibility was that additional teachers would try to transfer into the incentive schools in the
second year of the project. As part of the agreement between the Government of AP and the
Azim Premji Foundation, the Government agreed to minimize transfers into and out of the
sample schools for the duration of the study. The average teacher turnover in the second year
was only 5%, and once again, there was no significant difference in teacher transfer rates across
the various treatments (Table 1 – Panel B, rows 13 – 16).

10 Paul Johnson September 30, 2009 at 1:59 pm

It’s a very interesting paper but of limited application to the U.S. since primary (elementary) schools are not the major problem.

11 Ben September 30, 2009 at 2:44 pm

“the influence of the teacher on student performance is extremely minor” –tim

Really? I’m pretty sure that’s the opposite of true.

“classroom teachers are the most important factor in achieving gains in student achievement”
http://www.archindy.org/NCEA/files/press/Teacher%20Impact%20on%20Student%20Proficiency%20and%20Growth%20NCEA.doc
“measures of teacher preparation and certification are by far the strongest correlates of student achievement in reading and mathematics”
http://epaa.asu.edu/epaa/v8n1/
“TFA teachers tend to have a positive effect on high school student test scores relative to non-TFA teachers”
http://www.urban.org/publications/411642.html

12 Maximum Liberty September 30, 2009 at 5:28 pm

Paul Johnson says:
“It’s a very interesting paper but of limited application to the U.S. since primary (elementary) schools are not the major problem.”

Really? High school teachers certainly say that the kids they get from middle schools are often not up to grade level. I suspect that middle school teachers would say that they also get many unready students.

(I’m not saying that the paper necessarily is highly applicable to the US, just that this rationale for it not being highly applicable is off-base.)

Max

13 Seth September 30, 2009 at 6:07 pm

Arun – I’m not buying their logic on transfer rates of teachers. First, the rates don’t say say who transferred where. Perhaps the better teachers, transferred to the reward schools and other teachers transferred out, not wanting to face the extra accountability.

Also, the 33% transfer rate in the first year, which is “consistent with the
rotation of teachers once every 3 years” while the 5% turnover rate in the second year raises an eyebrow for me. Why isn’t the transfer rate in the second year not consistent with the “rotation every 3 years”?

14 gulzar September 30, 2009 at 10:50 pm

I am inclined to believe that the performance-based pay system being proposed may be more difficult to scale up than anticipated. While it may have succeeded in the limited context and time over which it was implemented, it may not yield the desired results with a more ambitious scope and larger area of implementation. As the program is expanded to cover all the teachers, the incentive award risks becoming an routine entitlement and getting diluted.

There are likely to be four major problems associated with the determination of bonuses. First, what would be the most optimum bonus? It should neither be too small as to have limited incentive effect nor too large as to waste scarce resources and generate incentive distortions. Second, as the program coverage expands, it will become difficult to cover all the teachers under a single bonus formula with broad acceptance among the stakeholders. Third, there will be increasing conflict between the search for accuracy of the formula in approximating the actual impact of teacher efforts and its transparency so as to generate broad-based acceptance. Finally, and most crucially, there will always be the problem with administering accurate data collection on such a massive scale. How do we guard against potential problems like grade inflation and data manipulation?

Efforts to incentivize teachers with performance based-pay and choice in transfers have been attempted in many states across India, albeit on smaller scales, with not so satisfactory results. The problem ultimately boils down to administering such massive programs and guarding against the dilution of its standards as it expands to cover all the schools. Given the standards of school supervision that prevails and the massive numbers of supervisors at different levels involved, it will be a major challenge to ensure the quality of data collection even for a district, leave alone a state. It will be very difficult, even with extensive computerization to effectively address this challenge. Once the programs are institutionalized, there is an ever-present danger that the performance incentive system loses its sanctity.

Further, in the absence of a sunset clause, these incentives are liable to be distorted. With time, there is the imminent danger, especially given the influence wielded by unions, that the incentives could become internalized as part of the salary structure. Ultimately, this could merely add to the substantial premium that government teachers enjoy over private school teachers.

the entire post is available here
http://gulzar05.blogspot.com/2009/09/designing-incentives-to-improve-teacher.html

15 LicketySplit October 2, 2009 at 6:47 pm

The whole debate about teacher pay being tied to performance really irks me. After all, it is the students performance being measured, and the teacher is being held accountable. How about this? Students be held accountable for their own performance? They are the ones that stand to lose or gain from their education. If the student feels the teacher isn’t teaching them well,or going to fast, or too slow for their needs, let them choose the teacher best for them. Teaching is simply a transfer of knowledge. We are getting to a point where formal institutions impede that transfer rather than facilitate it. This include everything from grammar school to the PhD level.

16 Adam October 6, 2009 at 1:41 pm

While I do not doubt that money can be a motivator, it can also be a dismotivator (counteracts non-financial motivators such as guilt, duty, etc.) when it is not of a sufficient amount to stimulate improvement.

Also, I would imagine that while performance on certain tests may go up, that learning might suffer in areas outside these tests as the focus shifts.

17 Brian May 4, 2010 at 8:11 am

teachers definitly need a raise. the average salary is not good. see http://www.salaryexplorer.com. hope that helps.

18 商家导航 October 22, 2010 at 1:57 am

商家导航

19 christmas messages November 22, 2010 at 2:26 am

Its civil servants are not known for their intrinsic motivation.” Err… do you mean India or U.S.? If both then the comparison of course is relevant.

20 phones wholesale December 2, 2010 at 2:55 am

NICE SITE and NICE PLACE!!! Wish to come and visit your place

21 estee lauder perfume December 20, 2010 at 6:37 am

Teaching is simply a transfer of knowledge. We are getting to a point where formal institutions impede that transfer rather than facilitate it. This include everything from grammar school to the PhD level.

22 岁月无痕 December 27, 2010 at 2:24 am

This has been a very controversial policy

23 岁月无痕 December 27, 2010 at 2:27 am

I love seeing websites that understand the value of providing a quality resource for free

24 岁月无痕 December 27, 2010 at 2:29 am

thank you for sharing such wonderful post

25 岁月无痕 December 27, 2010 at 2:32 am

thank you for sharing such wonderful post

thanks,thanks

26 岁月无痕 December 27, 2010 at 2:34 am

very interesting.

thanks

27 wholesale electronics January 8, 2011 at 3:54 am

In education, the standard for effectiveness is 0.25 standard deviations to be considered educationally significant. Below this threshold you’re unlikely to get an observable improvement. So, the study shows barely educationally significant results in math and educationally insignicant results in language arts.

28 Blinds and Shades January 11, 2011 at 5:09 am

Performance pay for teachers is frequently suggested as a way of improving education outcomes in schools, but the theoretical predictions regarding its effectiveness are ambiguous and the empirical evidence to date is limited and mixed. We present results from a randomized evaluation of a teacher incentive program implemented across a large representative sample of government-run rural primary schools in the Indian state of Andhra Pradesh. The program provided bonus payments to teachers based on the average improvement of their students’ test scores in independently administered learning assessments (with a mean bonus of 3% of annual pay). At the end of two years of the program, students in incentive schools performed significantly better than those in control schools by 0.28 and 0.16 standard deviations in math and language tests respectively. They scored significantly higher on “conceptual” as well as “mechanical” components of the tests, suggesting that the gains in test scores represented an actual increase in learning outcomes. Incentive schools also performed better on subjects for which there were no incentives, suggesting positive spillovers. Group and individual incentive schools performed equally well in the first year of the program, but the individual incentive schools outperformed in the second year. Incentive schools performed significantly better than other randomly-chosen schools that received additional schooling inputs of a similar value.

29 China Wholesale Store March 8, 2011 at 12:55 am

Teacher is the hard working. I appreciate them

Comments on this entry are closed.

Previous post:

Next post: