muralidharan

Online Education and Personalized Learning

Today I spoke at Brookings India on Online Education and India. One of the things I discussed was how online technology and AI can dynamically adjust content to the needs of an individual learner. An Indian firm, Mindspark, is a leader in mathematics education that is synchronized to an individual student’s actual ability regardless of grade. The ubiquitous Karthik Muralidharan with co-authors Abhijeet Singh and Alejandro Ganimian have an important paper doing a RCT on Mindspark, finding large gains in math ability and also in Hindi ability for students who win vouchers to the program. David Evans at Development Impact the World Bank blog has an excellent post on the Mindspark RCT.

I want to focus on a different issues: the personalization of education is especially important in India because classes often contain students of widely different abilities. Here’s a graph from Muralidharan et al. showing the student’s grade along the horizontal axis with the student’s actual ability on the vertical axis. The students are drawn from a sample of Delhi public schools.

Grades

The graph shows two things of importance. First, if most students were operating at grade level the dots/students would be clustered around the blue line. But very few students in grade 6 are operating at a grade 6 level–most are operating at a grade 3 or 4 level and some even at a lower level. The distribution of ability level in the same grade is extreme. No math teacher can be expected to teach students in the same class who are operating at grade levels 2-7. Even if the teacher teaches to the level of the average student the material will go over the heads of many. As a result, many students do not progress. Indeed, the second point is shown by the red line, the best-fit line for academic growth. The growth in achievement is slower than the growth in the standard. As a result, over time students fall further and further behind the standard.

Keeping all students in the same grade at a similar level of ability would be excellent and the best way to do this is by teaching to a student’s actual ability but the only way to do that on an economical basis is through online learning and AI technology.

Private versus Public Health Care in India

In an important paper in the latest AER, Das, Holla, Mohpal and the excellent Karthik Muralidharan compare private and public health care in India. (I once asked, “Is any economist doing more important work with greater potential for real improvement in the lives of millions than Karthik Muralidharan?” See previous posts on Karthik’s work for the answer.)

The AER paper examines health care in villages in Madhya Pradesh, one of the poorer states in India (GDP per capita of $1,500 PPP). In India, primary health care is ostensibly available for free from public health clinics and hospitals manned by professionally trained nurses and physicians. As with teachers at public schools, however, it’s very common for doctors at public clinics to be absent on any given day (40% were absent on a given day in 2010) and public clinics are not highly regarded. As a result, some 70% percent of primary care visits nationally–and an even higher percentage in Madhya Pradesh–are to private, fee-charging health-care providers. Most of the private providers do not have a license or medical degree although they may have some health-care training.

ruralhealthcareindiaThe authors sent trained actors, “standardized patients” to public and private clinics to evaluate provider effort and accuracy in response to the presentation of textbook symptoms of common illnesses (angina, asthma, and dysentery in a child at home). Standardized patients are used to train medical students in the United States and in India and the Indian SPs were trained by professionals including medical doctors, and a medical anthropologist familiar with local forms of presenting illnesses and symptoms.

The first result is that the provision of health care is uniformly and distressingly poor. Overall, only 2.6% of patients received a correct treatment (and nothing unnecessary or harmful). The private providers, however, exert much more effort than do the public providers. The private providers, for example, perform more items on a standard checklist and they spend more time with patients. But the private providers are no better than the public providers at giving a correct treatment. Why not?

Private providers exert more effort but are less knowledgeable. Loosely we might say that Quality=Effort*Knowledge. Private providers put in more effort but have less knowledge and public providers have more knowledge but put in less effort leading to similar quality levels overall.

There is one big difference, however, between the public and private regimes, the private regime is much less socially costly. Since costs are lower and the quality level is the same, the private system is much more productive. The authors note:

…our estimates suggest that the public health care system in India spends at least four times more per patient interaction but does not deliver better outcomes than the private sector

(FYI, this also holds true for public and private schooling in India and around the world. Private schooling is usually somewhat better or about as good as public schooling but much less costly so the productivity of private schooling is much higher.)

To focus on the issue of market incentives rather than knowledge the authors do a second set of remarkable tests. Indian doctors often work in a public and a private practice. Thus, the authors send standardized patients to the same doctors but in one case the patient is treated under the public regime and in other under the private, market regime. Once knowledge is controlled for the results are very clear, private, markets dominate the public regime.

…treatments provided in the private practice strictly dominate those provided in the public practice of the same doctor. The rate of correct treatment is 42 percent higher (16 percentage points on a base of 37 percent), the rate of providing a clinically non-indicated palliative treatment is 20 percent lower (12.7 percentage points on a base of 64 percent), and the rate of antibiotic provision is 28 percent lower (13.9 percentage points on a base of 49 percent) in the private practice relative to the public practice of the same doctor.

The bottom line is that the private market for health care is much bigger and less expensive than the public health regime in rural India and once we control for knowledge it’s of higher quality. These results have important implications for reform. In particular, much more effort should go into improving the knowledge of the private sector.

….the marginal returns to better training and credentialing may be higher for private health care providers who have stronger incentives for exerting effort. Current policy thinking often points in the opposite direction, with a focus on hiring, training, and capacity building in the public sector on one hand (without much attention to their incentives for effort), and considerable resistance to training and providing legitimacy to unqualified private providers on the other.

India’s biometric smartcards, a good sentence about them

Overall, our results suggest that investing in secure payments infrastructure can significantly enhance “state capacity” to implement welfare programs in developing countries.

That is from Muralidharan, Niehaus, and Sukhtankar in the latest American Economic Review.  Their main result is this:

We find that, while incompletely implemented, the new system delivered a faster, more predictable, and less corrupt NREGS payments process without adversely affecting program access.

Most of all there is lower leakage of benefits, and program participants strongly prefer the biometric arrangements and the accompanying direct cash transfers.  The measurements of this paper, by the way, are based on 19 million data points.

I believe the Indian biometric smartcard initiative remains under-discussed and underappreciated.  It is actually one of the greatest achievements of contemporary times, based upon the innovative mobilization of the labor of millions in a manner that probably only India could do and that at first sounded quite ridiculous.  Scan, record, and use the biometric information of over a billion people, and in a “backward” country at that.  Well, they haven’t finished but it is well on track to succeed.

I do worry about the privacy implications of the technology and the data collection, but as it stands today so many Indians don’t have that much privacy in any case.

Here are ungated versions of the paper.  Here is my earlier post on the paper and the technology.  I had written:

One broader lesson here is that developing nations are not merely copying and applying the inventions of the West, but innovating on their own.  But a lot of their innovations take labor-intensive rather than capital-intensive forms, and thus they do not always look like innovations to our sometimes ethnocentric eyes.

Still true.

India Fact of the Day

In India, for example, the number of taxpayers in relation to voters in the economy has been about 4-4.5% for a long time.

That is from an in-depth discussion about the Indian economy between Karthik Muralidharan and Arvind Subramanian (Chief Economic Adviser, Government of India). The reference is to income tax, of course. It’s a great discussion and the best place to begin if you want to understand the Indian economy today.

Low Cost Private Schools in the Developing World

Private schools for the poor are growing rapidly throughout the developing world. The Economist has a review:

PrivateSchoolingPrivate schools enroll a much bigger share of primary-school pupils in poor countries than in rich ones: a fifth, according to data compiled from official sources, up from a tenth two decades ago (see chart 1). Since they are often unregistered, this is sure to be an underestimate. A school census in Lagos in 2010-11, for example, found four times as many private schools as in government records. UNESCO, the UN agency responsible for education, estimates that half of all spending on education in poor countries comes out of parents’ pockets (see chart 2). In rich countries the share is much lower.

Overall, there is good evidence that private school systems tend to create small but meaningful increases in achievement (e.g. herehere, here, here) and especially good evidence that they do so with large costs savings. The large costs savings suggest that with the right institutional structure, which might involve vouchers and nationally comparable testing, an entrepreneurial private sector could create very large gains. Karthik Muralidharan who has done key work on private schools and performance pay in India puts it this way:

Since private schools achieved equal or better outcomes at one-third the cost, the fundamental question that needs to be asked is “How much better could private management do if they had three times their current level of per-child spending?”

The Economist notes that another promising development is national chains which can scale and more quickly adopt best practices:

…Bridge International Academies, which runs around 400 primary schools in Kenya and Uganda, and plans to open more in Nigeria and India, is the biggest, with backers including Facebook’s chief executive, Mark Zuckerberg, and Bill Gates. Omega Schools has 38 institutions in Ghana. (Pearson, which owns 50% of The Economist, has stakes in both Bridge and Omega.) Low-cost chains with a dozen schools or fewer have recently been established in India, Nigeria, the Philippines and South Africa.

Bridge’s cost-cutting strategies include using standardised buildings made of unfinished wooden beams, corrugated steel and iron mesh, and scripted lessons that teachers recite from hand-held computers linked to a central system. That saves on teacher training and monitoring.

The Economist is somewhat skeptical of scripted lessons, known as Direct Instruction in the education world, but in fact no other teaching method has as strong a record of proven success in randomized experiments (see also here and here).

Need I also point out that online education can bring some of the best teachers in the world to everyone, everywhere at low cost? An article in Technology Review titled India loves MOOCs points out that students from India are a large fraction of online students (fyi, we are also finding many Indian students at Marginal Revolution University)

Throughout India, online education is gaining favor as a career accelerator, particularly in technical fields. Indian enrollments account for about 8 percent of worldwide activity in Coursera and 12 percent in edX, the two leading providers of massive open online courses, or MOOCs. Only the United States’ share is clearly higher; China’s is roughly comparable.

Education is changing very rapidly and its the developing world which is leading the way.

Triple Differencing Bikes in India

In 2007 in an effort to increase the number of girls enrolled in school the government of Bihar in India gave each schoolgirl of age 14 a bicycle. The excellent Karthik Muralidharan and co-author Nishith Prakash set out to discover whether the program was effective. To jump to the conclusion they found that the program increased the enrollment of girls by 41% reducing the gender gap by almost half.

The reason for this post, however, is not the result–important as it is–but the two videos the International Growth Center made to explain Muralidharan and Prakash’s research methods. The first video explains the background of the research and then gives a very elegant explanation of triple-differences as an estimation strategy.

The second video explains that the researchers still weren’t completely happy that they had truly identified a causal effect (or perhaps the referees were not completely happy) so they hit on a complementary approach, looking for a dose-response relationship. With the collection of more data Muralidharan and Prakash were able to ask whether the program was more effective for the students who were neither so close nor so far from the school that a bicycle wouldn’t make a difference. Indeed, the program was most effective for students who lived at bicycle-relevant distances.

These videos are an interesting peek at some of the questions economists ask and the methods they use to answer those questions. The videos would be excellent for classroom use–challenge your students after the first video to come up with potential problems with the triple difference method and see if they can identify another research design that would address these problems!

Addendum: Here are previous MR posts on Karthik Muralidharan’s important research program.

How is the biomarker ID aid plan going in India?

One of the most important positive developments of our time – both underpublicized and underappreciated — is our growing ability to send and receive money securely across space.  It’s not just Paypal or Bitcoin in the West, as the truly significant gains from payment systems are coming in the developing world.  In particular, the efforts of the Indian government to set up a biometrically-based payments system are improving the lives of many millions and may go down as one of the most impressive achievements of contemporary times.

In 2009, the government of India set out to create unique, biometric-linked IDs for all 1.2 billion Indian citizens, based on fingerprints and a digital photograph.  Once the identities of these persons are tagged, the government will use the new system to deliver direct cash payments as a form of welfare aid.  To the extent the system works, programs with waste and leakage rates of 40% to 80% will become much more efficient.  Imagine instituting a direct cash transfer in lieu of a low productivity make-work job or sending welfare payments directly to beneficiaries rather than channeling them through corrupt local village officials, who take a cut off the top.

When the biomarker idea was proposed, it was far from obvious it would succeed.  The Indian government has failed at many basic tasks of infrastructure, such as good roads or clean water, and in general the quality of governance is not reliable.  Furthermore conditions in India seemed less than ideal for such an endeavor, as for instance about half of India does not have even a bank account.

There is now a major formal study of how well this new program is going and the results are strongly positive, as shown in “Payments Infrastructure and the Performance of Public Programs: Evidence from Biometric Smartcards in India,” a new NBER Working Paper by Karthik Muralidharan, Paul Niehaus, and Sandip Sukhtankar (ungated copies here).

The authors look at one Indian state, Andhra Pradesh, and rely on a large-scale experiment which gave some people the new service and others not, on a randomized basis.  The results are impressive.  The average household was able to receive 23% more in aid, and more quickly, while the government’s rate of “leakage” – lost or misdirected aid – declined by over 12%.  Overall the new method cost no more than the old, and there were no additional problems of access.  The authors estimate that the benefits in time savings to beneficiaries, taken alone, are larger than the costs of creating the new payments system.  For poor people, those gains represent major life improvements.

No less importantly, the beneficiaries strongly favored the new method of aid by margins of eighty to ninety percent.  That means a recent Indian Supreme Court decision, ruling against making the new system mandatory for privacy-related reasons, is unlikely to stop its ultimate success.

Despite many obstacles and imperfections, the logistics of the system seem to be coming together.  After two years of roll out, the share of Smartcard-enabled payments in the relevant studied districts is running at about fifty percent.  It now seems plausible to imagine that most eligible Indian citizens are in some way connected to the system within the next ten years.  Liberals may prefer to think of this as a boost in “state capacity,” whereas conservatives can see it as a paring back of government programs which were not working and as replacing corrupt and paternalistic in-kind aid with direct cash transfer, as had been suggested by Milton Friedman.

The nature of this Indian innovation has been the combination of modern (but not cutting edge) information technology with the use of labor on a very large scale for implementation.  The process of registering so many Indians, and recording their biodata, has required the mobilization of an immense army of labor in a manner which is only possible in a low-wage country, albeit one with a fairly active bureaucracy.

One broader lesson here is that developing nations are not merely copying and applying the inventions of the West, but innovating on their own.  But a lot of their innovations take labor-intensive rather than capital-intensive forms, and thus they do not always look like innovations to our sometimes ethnocentric eyes.

China too may be a more innovative nation than it at first appears.  Sometimes the Chinese contribution to a production process is dismissed as merely adding to a single stage of production, such as finishing off an iPhone to be shipped out.  The deeper truth is that China offers not only cheaper wages but also a very large pool of skilled workers, including engineers, which can be mobilized in large numbers with extreme rapidity.  To create such a talented labor pool on such a scale is an unprecedented innovation and it is one which the West has not managed to match.

The bottom line is that today I have good news to report.

Private Schooling In India: Results from a Randomized Trial

Karthik Muralidharan runs very large, randomized controlled trials on education in India. His previous work showed that performance pay for teachers in India has large and significant improvements on student learning. In his latest paper (with Venkatesh Sundararaman) he reports on the results of The Andhra Pradesh School Choice Project, a long-term randomized controlled trial covering over 6,000 students in 180 villages for four years (2008-2012). The study offered students a lottery for a private school scholarships and lottery winners were compared with non-winners. The results show modest improvements in learning for private school students and big increases in school productivity.

We find that private school teachers have lower levels of formal education and training than public-school teachers, and
are paid much lower salaries. On the other hand, private schools have a longer school day, a longer
school year, smaller class sizes, lower teacher absence, higher teaching activity, and better school
hygiene. After two and four years of the program, we find no difference between the test scores of
lottery winners and losers on math and Telugu (native language). However, private schools spend
significantly less instructional time on these subjects, and use the extra time to teach more English,
Science, Social Studies, and Hindi. Averaged across all subjects, lottery winners score 0.13 σhigher,
and students who attend private schools score 0.23 σhigher. We find no evidence of spillovers on
public-school students who do not apply for the voucher, or on students who start out in private schools
to begin with, suggesting that the program had no adverse effects on these groups. Finally, the mean
cost per student in the private schools in our sample is less than a third of the cost in public schools.
Our results suggest that private schools in this setting deliver (slightly) better test score gains than
their public counterparts, and do so at substantially lower costs per student.

As Karthik notes in a Ideas for India short article that summarizes:

Since private schools achieved equal or better outcomes at one-third the cost, the fundamental question that needs to be asked is “How much better could private management do if they had three times their current level of per-child spending?”

Is any economist doing more important work with greater potential for real improvement in the lives of millions than Karthik Muralidharan?

Shout it from the Rooftops! Performance Pay for Teachers in India

Several years ago I reported on a very large, randomized experiment (JSTOR) on teacher performance pay in India that showed that even modest incentives could significantly raise student achievement and do so not only in the incentivized subjects but also in other non-incentivized subjects, suggesting positive spillovers. The earlier paper looked at the first two years of the program. One of the authors, Karthik Muralidharan, now has a follow-up paper, showing what happens over 5 years. The results are impressive and important:

Students who had completed their entire five years of primary
school education under the program scored 0.54 and 0.35 standard deviations (SD) higher than
those in control schools in math and language tests respectively. These are large effects
corresponding to approximately 20 and 14 percentile point improvements at the median of a
normal distribution, and are larger than the effects found in most other education interventions in
developing countries (see Dhaliwal et al. 2011).

Second, the results suggest that these test score gains represent genuine additions to human
capital as opposed to reflecting only ‘teaching to the test’. Students in individual teacher
incentive schools score significantly better on both non-repeat as well as repeat questions; on
both multiple-choice and free-response questions; and on questions designed to test conceptual
understanding as well as questions that could be answered through rote learning. Most
importantly, these students also perform significantly better on subjects for which there were no
incentives – scoring 0.52 SD and 0.30 SD higher than students in control schools on tests in
science and social studies (though the bonuses were paid only for gains in math and language). There was also no differential attrition of students across treatment and control groups and no
evidence to suggest any adverse consequences of the programs.

…Finally, our estimates suggest that the individual teacher bonus program was
15-20 times more cost effective at raising test scores than the default ‘education quality
improvement’ policy of the Government of India, which is reducing class size from 40 to 30
students per teacher (Govt. of India, 2009).

In another important paper, written for the Government of India, Muralidharan summarizes the best research on public schools in developing countries. His conclusion is that there are demonstrably effective and feasible policies that could improve the public schools thereby increasing literacy and numeracy rates and raising the incomes of millions of people.

The generation entering Indian schools today is the largest that has ever, or for the foreseeable future, will ever enter Indian schools so the opportunity to raise educational quality for essentially the entire Indian workforce over the next several generations is truly immense.

Education in India

This is related to our recent discussion of why Indian test scores why so low:

Estimating the precise enrollment of private schools is tricky. Government officials say more than 90 percent of all primary schools are run by or financed by the government. Yet one government survey found that 30 percent of the 187 million students in grades 1 through 8 now attend private schools. Some academic studies have suggested that more than half of all urban students now attend private academies.

In Mumbai, so many parents have pulled their children out of government schools that officials have started renting empty classrooms to charities and labor unions — and even to private schools. In recent years, Indian officials have increased spending on government education, dedicating far more money for new schools, hiring teachers and providing free lunches to students. Still, more and more parents are choosing to go private.

“What does it say about the quality of your product that you can’t even give it away for free?” Mr. Muralidharan said.

Here is much more.

Teacher Performance Pay: Experimental Evidence from India

In an impressive new paper, Karthik Muralidharan and Venkatesh Sundararaman provide evidence on the power of teacher incentives to increase learning.  The paper is impressive for three reasons:

1) Evidence comes from a very large sample, 500 schools covering approximately 55,000 students, and treatment regimes and controls are randomly assigned to schools in a careful, stratified design. 

2) An individual-incentive plan and a group-incentive plan are compared to a control group and to two types of unconditional extra-spending treatments (a block grant and hiring an extra teacher).  Thus the authors can test not only whether an incentive plan works relative to no plan but also whether an incentive plan works relative to spending a similar amount of money on "improving schools."

3)  The authors understand incentive design and they test for whether their incentive plan reduces learning on non-performance pay margins.

The results are as follows:

We find that the teacher performance pay program was highly effective in improving student
learning. At the end of two years of the program, students in incentive schools performed
significantly better than those in comparison schools by 0.28 and 0.16 standard deviations (SD)
in math and language tests respectively….

We find no evidence of any adverse consequences as a result of the incentive programs.
Incentive schools do significantly better on both mechanical components of the test (designed to
reflect rote learning) and conceptual components of the test (designed to capture deeper
understanding of the material),suggesting that the gains in test scores represent an actual
increase in learning outcomes. Students in incentive schools do significantly better not only in
math and language (for which there were incentives), but also in science and social studies (for
which there were no incentives), suggesting positive spillover effects….

School-level group incentives and teacher-level individual incentives perform equally well in
the first year of the program, but the individual incentive schools significantly outperformed the
group incentive schools in the second year….

We find that performance-based bonus payments to teachers were a significantly more cost
effective way of increasing student test scores compared to spending a similar amount of money
unconditionally on additional schooling inputs.

Surprisingly, since absent teachers are a big problem in India, reduced teacher absenteeism per se does not appear to be the primary mechanism by which incentives improve learning.  Instead the primary mechanism appears to be more intensive teaching, including more homework and classwork and better attention to weaker students, this greatly increases the relevance of these results to teaching in the developed world.

Addendum: See also Karthik's comments on the comments at 26.