Online Education and Personalized Learning

Today I spoke at Brookings India on Online Education and India. One of the things I discussed was how online technology and AI can dynamically adjust content to the needs of an individual learner. An Indian firm, Mindspark, is a leader in mathematics education that is synchronized to an individual student’s actual ability regardless of grade. The ubiquitous Karthik Muralidharan with co-authors Abhijeet Singh and Alejandro Ganimian have an important paper doing a RCT on Mindspark, finding large gains in math ability and also in Hindi ability for students who win vouchers to the program. David Evans at Development Impact the World Bank blog has an excellent post on the Mindspark RCT.

I want to focus on a different issues: the personalization of education is especially important in India because classes often contain students of widely different abilities. Here’s a graph from Muralidharan et al. showing the student’s grade along the horizontal axis with the student’s actual ability on the vertical axis. The students are drawn from a sample of Delhi public schools.


The graph shows two things of importance. First, if most students were operating at grade level the dots/students would be clustered around the blue line. But very few students in grade 6 are operating at a grade 6 level–most are operating at a grade 3 or 4 level and some even at a lower level. The distribution of ability level in the same grade is extreme. No math teacher can be expected to teach students in the same class who are operating at grade levels 2-7. Even if the teacher teaches to the level of the average student the material will go over the heads of many. As a result, many students do not progress. Indeed, the second point is shown by the red line, the best-fit line for academic growth. The growth in achievement is slower than the growth in the standard. As a result, over time students fall further and further behind the standard.

Keeping all students in the same grade at a similar level of ability would be excellent and the best way to do this is by teaching to a student’s actual ability but the only way to do that on an economical basis is through online learning and AI technology.


So you are saying that modern teachers would be incompetent to teach in, say, the one-room schoolhouse
(of my grandparents' and even my parents' generation)?

It was reading, writing and arithmetic.

And not functions and graphs.

Variables, functions, and graphs are the transition from arithmetic to pre-algebra.

Very good point. My 7th grader is learning far more complex math than I was at that age.

Rote learning should last until abstract learning (13 yo) could take place.

They used different techniques -- the "modern" stand-in-front-of-the-class-and-lecture method is particularly ill-suited. But also -- they were hearing the more advanced material from the beginning. This is different -- the assumption that everyone in 7th grade (say) needs to learn 7th grade stuff is baked in.

That describes my experience as engineering undergrad. Ability & interest distribution was extreme. Courses were boring some times because the teacher was teaching to the level of average student. Other times people with sports scholarships were sabotaging the course. Learning happened during appointments at teacher's office to deal with specific questions: "I listened to you, I read the book but I don't understand THIS".

Online learning and AI technology may help. But there's an ancient method: repeating a grade. There are tons of research that show repeating a grade may not be good for the retained student. However, there's almost no research on the benefits on all other students when low achieving ones are retained. This plot would look nicer if the students with two lower assessed grades are retained. This is unpopular because graduation rates go down, but what's the point of having a graduation rate of 100% if students have no knowledge?

>...repeating a grade...

That is a blunt instrument. Self-paced online ed is more efficient.

Summer Vacation, the effect of 1840s rich slave owners wanting to enjoy their summer homes.

Farm Kids, December to March, and May to August are the two terms of schooling.

Back in the days, pre-1840, urban schools opperate every weekday. School still holds on holidays since it makes sense to celibrate it together. Culture is stronger by having holidays together as a community.

Especially if the weakness may only be in one critical area.

Studies show that year round schools with 180 school days only decrease the standard deviation of performance. Low performing students do not burn out as much. High performing students lose their momentum. Both group of students perform closer to average.

This is fascinating -- do you have a reference? I would love to see the study.

Most Year Round School experiments focus on the low performing students. Got to look at average performance and standard deviation comparison.

I forgot which study it was, but it has to do with the one that looks at "12-month" performance and not "beginning of summer and end of summer" (intentionally bias towards YRS) performance.

Another bias is that YRS with sufficient extra money would give additional support classes to poor performing students instead of giving them vacation. Thus, they get more than 180 days of school. (My Hypothesis) Those poor performing students may have low attendance in the first place, so those additional days only make up so losses.

Disagree. Evidence suggests that summer vacation widens the gap between rich and poor students.

The rich kids get intellectually stimulating summer camps, the poor kids get into trouble or just watch TV.

"Research indicates that, on average, students lose skills over the summer, particularly in mathematics. However, not all students experience “average” losses, and summer learning loss disproportionately affects low-income students. Low-income students lose substantial ground in reading during the summer, while their higher-income peers often gain. Most disturbing is that it appears that summer learning loss is cumulative and that, over time, these periods of differential learning rates between low-income and higher-income students contribute substantially to the achievement gap in reading. It may be that efforts to close the achievement gap during the school year alone will be unsuccessful."

Great selection bias. Since most schools have vacation in the summer, blame the summer. In fact, 180 days of school regardless as to how it is arranged, tends to give the same results. Only by increasing to 240 (no school on holidays) to 260 (no breaks on holidays) school days would close the gap. However, this requires teaching new materials, not dilute the materials. Look at 96 months of learning below, why does 8 year grammar school is enough for college back in the 19th Century!

In the 19th century, 8 years of grammar school is enough to prepare stusents to college.
In the 20th century, the secondary education reform (1920s) made high school a prerequisites of college. Before the reform, high school is trade school and vocational school. Now, trade schools and vocational schools are post-secondary education.

It is the same amount of information. Before 1840s, students learn 12 months for 8 years. After 1920s, students learn 8 months (+1 review month) for 12 years. Both counts to 96 months of learning.

It made more sense for the Baccalaureate (the gender neutral term for Bachelor's Degree (Men) or Bachelorette's Degree (Women)) to occur at 18 years old since it represents "unmarried" adult.

The Traditional Apprenticeship of 4 years to Baccalaureate, 3 years to Masterate (the gender neutral term for Master's Degree (Men) or Mastress' Degree (Women)) coincides with [b]18 years old[\b] and [b]21 years old[\b] for a purpose.

Note 1: Mastress (unmarried women, Mstrs.) merged with Mistress (married women, Mrs.) because Mastress looks/spells too similar to Mattress.

Note 2: Master (unmarried men, Mstr.) and Mister (married men, Mr.) have evolved. Master moved from unmarried men to (young) boys (of wealthy backgrounds), while Mister changed into men ignoring marital status.

Doctorate is the gender neutral term for Doctor's Degree (Men) and Doctress' Degree (Women).

Personalized online education, designed to fit the ability and level of knowledge of the individual student, would revolutionize education. No longer would education have to be targeted to the mythical average student. Tabarrok mentions targeting education to the low achiever, but perhaps more importantly it could be targeted to the high achiever who is often bored with instruction targeted to the mythical average student. But that's just one of many possibilities for targeting online education. To name a few, online education could be targeted to English speaking and Spanish speaking, wealthy students and poor students, young students and old students, liberal students and conservative students, fanatic students and moderate students. No longer would education be targeted at the mythical average student, no longer would education serve as the great social and political leveler. Average would indeed be over. Of course, it also means society would be much more divided, conflict much more accentuated, the individual emphasized over the community. What could go wrong?

Adult Americans already can have all the Liberal or Conservative information and instruction they (or sponsors or the government or NGOs or churches) are willing to pay. As for K-12, school boards are hotly discussing if evolution happened in seven days and if Senator McCarthy was an unsung hero and why he left the Beatles. It is not clear to me that students benefit from unified stupidity. As Mr. Mao Zedong famously said, "let a hundred flowers bloom."

to be noted: "let a hundred flowers bloom" is a set-up to identify liberal intellectuals.

It would be good to see the experiment on how well Ai does in a school population of mixed abilities. You have to also consider that education is a social activity...students learn from each other (not just a computer), or, in some cases they learn bad habits from each other. Classroom have internal externalities. Also, there is a matter of efficiency as well in changing the person who is far behind his grade level.. An eighth grader reading at a 5th grade level is harder to change, and also missed out on other reading related learning from grades five to eight. Maybe the answer is starting AI at grade 3 and not letting the deviations from the expected norm arise, such as using AI earlier and creating a summer AI product or program, having these kids catch up when school is closed as well.

Roland Fryer, an economist at Harvard, did an experiment using intense remedial tutoring in some Houston schools at grades 3, 6, and 9, and found that it, along with some other changes (such as lengthened school years and school days, and greater time on task in classrooms) resulted in marked improvement in student test scores. So it isn't exactly the same as using AI for remedial education, but it does demonstrate that regularly providing effective remedial education can work. And while it is likely more expensive to use tutors and to pay teachers more, I found it surprising that it only cost in the range of $2-3k more per student, which given the improvement in outcomes, seems like a reasonable amount of money to spend.

Also, Fryer is the economist who believes that his work shows that most of the income and employment gap between blacks and other Americans is due to differences in education between blacks and other Americans. So Fryer believes that implementing the school reforms that he has studied would largely gaps in wages and unemployment levels between blacks and other Americans. If that is true, it seems that a cost of $2-3k per student per year is a bargain. And given that the schools Fryer studied were among the lowest performing schools in Houston, the per student cost throughout the nation would likely be lower than what was spent in those Houston schools.

Rich black kids score about as well as poor white kids, last I heard.

Check out Reasoning Mind. It's a non profit based in Houston that has developed AI-based math education. It is delivered via computers in the classroom. I used it in my classroom and found it awesome. Students like it and the results have been quite good.

Tim Taylor has a good summary of the current state of the Indian economy. In particular, he points out the divergence in skills across regions. Online education might be the magic bullet to achieve some degree of convergence. [I am a regular consumer of online education and, accordingly, I am a supporter of it. My earlier comment isn't meant to diminish the importance and promise of online education, but highlight the potential for unintended consequences. As I've commented many times, when I was a child, long ago, average wasn't mythical; indeed, with few exceptions, everyone was average. Of course, that wasn't surprising since I attended segregated schools. In those days, schools really did serve a socializing function. Today, not so much, as schools and the students who attend them have segregated, de facto segregation today far greater than the de jure segregation of my childhood.]

From the Mindspark site:

"A world class Maths experience just at 3 cent a day", or $125 for a 12-month plan

Can't make this stuff up

Where do you see that? From, I see "A world class Maths experience just at 12 rupee a day", and their 12-month plan costs 4483 rupee -> which does make the daily cost around 12.3 rupees.

$0.03 per day * 365 days per year = $10.95 per year < $125.00 per year

Better to pay daily.

India just seems to put the social component of schooling ahead of the book learning component: It's something that in the US it's most often faced by parents with very high performing kids.

When looking into local elementary schools here in the US, my son was often tested, and some schools went as far as to recommended starting him in 3rd grade: He'd get lessons closer to his skill level, but he'd then get the social challeges of being much younger and smaller than his classmates. On the other end, taking him to the public school and putting him in grade, would mean a class that is focused on teaching letters and numbers to a kid that can multiply and do fractions.

I have high hopes that computer-guided education will make it easier for schools to have kids learn at the speed that is right for them, but I wonder how well the socialization aspect will go when the kids in a class don't share work, and the teacher becomes just a cheerleader.

When I complained about this in my fairly ordinary Massachusetts high school, my department chair said, "Teach to the 25th percentile."

The Top 8% only needs 120 school days to complete a grade level.

The Middle 84% needs 180 school days to complete a grade level.

The Bottom 8% needs 240 school days to complete a grade level.

“Teach to the 25th percentile” is still better than forcing the top students to learn at half speed they could handle.

The big problem with online education is motivation. It's one thing when everyone in the room is doing the same thing. But when you're sitting alone in front of a computer... Some people will love it and race ahead. But most kids, not being intrinsically interested in the material in the first place ...

True, but with enough options everyone can find something they are passionate about. They will persue the skills they need when they need them.

We have all been biased by an education model that is hundreds of years old, no longer needed, and obsolete.

A buffet of self-paced learning options leveraging animation, graphics, AI, and gaming is the future.

Anecdote: my 15 yo son took my iphone out of my hands to try a Chinese language learning app I was using and wouldn't give it back. Learning can be gamified!

...but with enough options everyone can find something they are passionate about.

True, but most of those things are not considered properly educational. There is a fairly limited menu of what students are allowed to choose from. Every state has standards that say, "You have to take this, and this, and this."

I would be willing to bet large amounts of money that "animation, graphics, AI, and gaming" will NOT transform things students are not interested in to things that they will eagerly sit down in front of a computer for.

+1 It's not the lack of algebra that holds back the worst students. The one thing that the bottom 10-15% need is discipline or good work habits. The ability to complete assignments on time, to come to class regularly, and to work with others without being disruptive. That's it.

This is the group that is most likely to fail even the most basic McD type jobs. Yet authoritarian style teaching that instills discipline is the one thing that modern public schools will not do and that shifting to online ed will only make worse.

But the evidence is that even lower end Catholic schools do succeed with this group (cf. Altonji's research).

"classes often contain students of widely different abilities": why? It's not God-given, you know.

You might as well quantify the penultimate sentence of your opening paragraph: The growth in achievement is about one third the growth in the standard. It takes three years for the pathetic dopes to achieve one year of standard progress.

I'm confused by the graph. Does this mean that students are assessed at a level of 4.5 (give or take a bit) or 4.0 (give or take a bit), but not as 4.2 (give or take)?

I'm happy to be directed to the appropriate a.i. adapted statistics course.

Actually, ignore that - suppose clusters around integer grade levels makes sense.

Clustering around integers makes sense for K-12. Once goes to post-secondary, you would see clustering around quarters ("thirds") and semesters (halves).

"AI" is not needed, except as a marketing term to sell good ideas to bureaucrats. Simple ability testing is fine, and has been available forever.

Abjlity testing correlates with status. The poor perform lower than the rich. It is best to create two extra grades for the poor, but then they would complain graduating high school at 20 instead of 18.

There was a two grade performance difference between the North and the South during the late 19th century and early 20th century. The modern gap is smaller, but back then, it was common for students transferring from South to North to repeat grades.

This squares pretty well with what I remember of my math education.

Math was always different from most of the other subjects in that there was a need to understand one module before advancing to the next. As students had differing intrinsic capabilities (for whatever reason), the variance of the knowledge distribution grew year-over-year.

I recall being pretty bored in class until I was old enough to be switched into advanced sections. On the other hand, a lot of students just didn't have the pre-requisites to succeed at the courses they were in. This made them fall behind further and associate very negative feeling with math. It's a shame. Fast forward to university-- most students don't study any math at all all, while others go very deep.

I really liked the opportunity to pursue more and more advanced mathematics in college. I thought I was kind of special until I started attending conferences as a graduate student. At that point, I was literally toe-to-toe with some of the best mathematical minds on the planet. That was just another level that I was clearly beneath.

I elected to quit academic math and become a Wall Street quant. I still get to do math and get the satisfaction of deploying systems in the real world. Oh and the pay is muy better. =)

AI and math set me looking for individual, computer help under a teacher's broad supervision.

Assuming student motivation and reliable testing, AI help with problem sets in math resembles AI scripted phonics (probably the best backup in case "whole language" alone does not make an individual thrive at reading).

Skype or such linking tutors with Khan Academy math problem sets seems the limiting factor expanding beyond 1 or 2% completion rates. Describing "how to solve the problems and then help learn the 'how'" must drive teaching coaches and an algorithm.

In US public schools, very low teacher math skills probably limit any success teaching math -Coleman 2 - so an AI coaching individuals plus math-competent volunteers might be a game changer. Individual instruction would be a fig leaf for unions.

"No math teacher can be expected to teach students in the same class who are operating at grade levels 2-7."

hahahahaha. Um. That's the life of a high school math teacher, particularly in a socioeconomically and racially diverse school.

I'm teaching Trig to kids from "kind of sorta grasps basic arithmetic" through "grasped 80% of second year algebra". I routinely teach Algebra 2 to kids who never understood algebra 1. I give my algebra 2 kids a test on the first day of school, a test designed to assess algebra 1 readiness for kids who just finished pre-algebra. Every year, the average wrong is about 10, but at least 5 kids in a class of 30 get 25-30 wrong.

And none of your solutions will work. What might work is if people like you started to grasp what happens in American schools, ever day, without your absurd notions of inept social workers who show movies all day in math class.

Better rural road networks would improve transportation access to the extent that, for practical purposes, a larger pool of students per physical unit of education facility location would enable more flexibility to have content, etc. matched.

Maybe there's an interesting tech fix for this. It would reduce the benefits of improved transportation infrastructure, and this kind of way of biasing future cost/benefit calculus should also be considered in advance.

Educated rural children will still need to get rural products to market, and in the meantime, the intermittant access to electricity, may constaint layouts. However, as opposed to the West, it strikes me as though Indian officials are extremely open to self-sufficiency ideas and ideals, for example to like the idea of a rural community being able to satisfy electricity needs without access to the grid, whereas I think this is a generally unpopular concept at the systems level in many places, where for some reason the government seems almost opposed to the idea of localized self-sufficiency in relation to something like energy.

SJWs dont like streaming, they think all students are equal. The amount of student streaming can be seen from the OECD PISA project which gives the distribution of the 15 yo students with respect to their grade, e.g. in Japan 100% are in grade 10, UK and SG have little streaming. What is interesting is that another OECD TALIS project surveyed the teachers at the coal face what they did. The teachers world wide do not seem to agree with those SJW.

TT2G42C: I give different work to the students who have difficulties learning and/or to those who can advance faster.

1. Never or almost never.
2. Occasionally.
3. Frequently.
4. In all or nearly all lessons.

Interestingly countries like UK where there are little streaming at the national/school levels, high pct of teachers were forced to practice it in the classroom, thus increasing their teaching loads.


64.02 494 11.8 21.8 95.02 #GBR

21.97 495 12.9 22.4 66.57 #FRA

33.25 519 15.3 12.3 84.97 #FIN

37.21 481 8.8 25.8 71.21 #USA

45.41 391 0.8 67.1 42.02 #BRA

21.65 536 23.7 11.1 100.0 #JPN

where Pct42C: pct of teachers who streamed the class

Math12 mean national PISA Math score

Mtop smart fraction

Mlow challenged fraction

PctGrade pct of 15 yo in the major grade

JPN, SGP and KOR seemed to be outlier as their mean Math scores were high, so presumely the challenged factions could follow the syllabus.

Interestingly, Pct42C also negatively correlated with cheating in class. But that is another story.

The Pct42C model,

Estimate Std. Error t value Pr( gt |t|)

(Intercept) 31.3217 8.3743 3.740 0.000876 ***

MTop -0.9852 0.2079 -4.740 6.13e-05 ***

PctGrade 0.2835 0.1074 2.639 0.013640 *
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

n=30; Rsa=0.4889; p=0.0001162

The dominant factor that determines why the teachers streamed the class, the smart fraction Mtop, i.e. the lower the value of Mtop the higher the pct of teachers streaming the class to help the challenged factions. This is totally the reverse of what the SJWs were asserting.

Education is always very important because it makes hell of a lot difference to anything. If we have enough education then only we stand bright chance to achieve better results or else we will only end up nowhere. I do Forex trading and I have gained a lot of education for it especially with help of OctaFX broker and with their amazing educational setup which covers all the parts and regularly gives news update, it’s absolutely fantastic for any newbie.

Comments for this post are closed