I, Robot?

by on May 22, 2012 at 6:35 am in Education | Permalink

In experiments at six public universities, students assigned randomly to statistics courses that relied heavily on “machine-guided learning” software — with reduced face time with instructors — did just as well, in less time, as their counterparts in traditional, instructor-centric versions of the courses. This largely held true regardless of the race, gender, age, enrollment status and family background of the students.

Here is more.  The report was led by William Bowen, an economist who is famous for, among other things, having described education as subject to an inexorable “cost disease” for lack of labor-saving innovation.

dearieme May 22, 2012 at 7:37 am

“machine-guided learning” might seem naturally compatible with the American tradition of “machine-guided examining”, by which I mean tick-box examination papers. It would be fascinating if “machine-guided learning” were also superior for different examining traditions.

Gunnar Tveiten May 22, 2012 at 8:28 am

It depends a lot on the subject – this was statistics. Machine-learning is very efficient for subjects that are clear and logical and with obvious correct or wrong answers. They work a lot less well for more subjective subjects.

For the subjects where it works, the benefits are many though. Individual pace. Infinite repeatability of needed lectures. Test understanding immediately. Show how the different parts fit together.

Those who ain’t tried it, should go try out khanacademys machine-learning for maths. They’ve got math from “count to 10″ up to some college-level stuff covered pretty well, and it’s well worth it to invest an hour or two into testing it out, even if you know math already. (but there’s something new for most people there)

Rahul May 22, 2012 at 11:38 am

I’m curious, what class level and which quality of university? If they chose a 101 class at a mediocre university I’m not surprised.

Echows May 23, 2012 at 3:23 am

Robots used to grade essays: http://www.nytimes.com/2012/04/23/education/robo-readers-used-to-grade-test-essays.html?_r=1

“Computer scoring produced “virtually identical levels of accuracy, with the software in some cases proving to be more reliable,””

Granted, it’s not perfect yet, since:
“The e-Rater’s biggest problem, he says, is that it can’t identify truth”,
but we’re definitely getting there.

choncan May 22, 2012 at 8:38 am

dearieme, are you writing satire? I went to an American university and took many math courses and I never once encountered a “tick-box examination paper”. Math courses both here and in the UK seem incredibly uniform in the sense that they progress through assigning problem sets consisting of, you know, math problems, then have tests where you are expected to work out — wait for it — math problems. What are you talking about? Did you take an undergraduate math course with some kind of individual oral examination? An essay test? (“How I feel about the Poisson process…”) Help me out.

Tracy W May 22, 2012 at 8:43 am

Choncan, perhaps Dearime is thinking of the difference between maths exams that only look at whether you get the right answer or not, versus maths exams where you can get partial marks for works, even if your final answer is wrong, (or exams where you’re given the answer and asked to work out the proof.)

Robot Instruction, Human Graders May 22, 2012 at 8:58 am

In those cases, you could have human graders and robot instructors.

jpa May 23, 2012 at 1:46 am

actually, robot graders would be more fair and consistent. grading math problem isn’t that hard of a computer science problem. they already grade english essay’s as well as humans.

Finch May 22, 2012 at 10:55 am

It’s pretty rare to get a math exam like you describe past high school, right? Certainly you never see it if you study a technical field like science or engineering. I think Dearime just has a mistaken impression of American education.

The only times I’ve had math exams like that past 10th grade they were standardized tests or certification exams, and I’ve only done a handful of those. The SAT and the GRE are like that, but they aren’t typical math exams.

RM May 22, 2012 at 1:29 pm

As someone who teaches a class that extensively uses simple math, the more pertinent question to me is “how many students”. This class does not have a TA. You can see where I am going with this. The problem is not confined to “math” classes only. In a 200-person intro class on any subject, without an extensive army of TAs, the grading will be correct/incorrect done by scantron.

TallDave May 22, 2012 at 9:21 am

Tuition has gone way, way up (K-12 expenses as well) even as non-institutional learning has become cheaper and more accessible. It’s hard not to see this as rentseeking.

Right Wing-nut May 22, 2012 at 11:40 am

Education is not lacking at all in innovation. It’s just that the innovation is aimed at consuming the Guaranteed Student Loans that the feds have so lovingly provided, at the college level, and the state budget at the state level.

What is the incentive for “educators” to innovate themselves into irrelevance?

Rich Berger May 22, 2012 at 11:13 am

The clamor for smaller class sizes is equivalent to a call for lower productivity.

Right Wing-nut May 22, 2012 at 11:41 am

On the contrary, it is a call to improve the quality of the output. Speeding up the belt doesn’t make for better sausage, although it may make for more profits.

Rich Berger May 22, 2012 at 11:49 am

I don’t see any evidence that the quality of the output has increased. The cost certainly has – lower productivity.

JWatts May 22, 2012 at 6:32 pm

The studies I’ve seen seem to indicate that lower class size has had very little effect on quality. Teacher quality has a much greater effect than class size. Shrinking class size is pretty much a payoff to the teachers union and has little detectable benefits. I’d much prefer increasing class size by 40 to 50% (from 20 to 30) and increasing teacher pay correspondingly to attract higher quality teachers.

doctorpat May 22, 2012 at 11:27 pm

Smaller class size automatically lowers the average quality of the teacher. The net effect may well be negative.

hamilton May 22, 2012 at 11:24 am

Golly; sure am glad I got that PhD so I could spend my days teaching undergraduates, which was my only real interest. If you’ll excuse me, I’ll just go kill myself now and get it over with…

Right Wing-nut May 22, 2012 at 11:43 am

Yeah, I figured that one out & took a terminal master’s. Sad thing is, now no one thinks I’m capable of PhD work. Blehh.

Donald A. Coffin May 22, 2012 at 11:34 am

A couple of points. It’s William BAUMOL, not William BOWEN, who is usually credited with the cost-disease argument.

And this was a hybrid stat class, which incorporated a computer-based learning environment with face-to-face (1 hour per week) meetings with instructors.

Also, I looked at the cost simulation material (it’s about two links into the chain), and I’m not convinced that there are really any cost savings here. YMMV.

Erik Brynjolfsson May 22, 2012 at 5:47 pm

For the record, both William Bowen and William Baumol have discussed the “cost disease”. See e.g. Baumol, William J. and William G. Bowen, Performing Arts: The Economic Dilemma, (Twentieth Century Fund, 1966).

Amasa Amos May 22, 2012 at 1:01 pm

There is something so appropriate about the idea of randomly assigning students to statistics courses.

Paul Johnson May 22, 2012 at 1:12 pm

My biggest problem is that in practice most of the learning technology is produced by publishers who design it so as to force students to buy their textbooks.

Wade May 22, 2012 at 4:54 pm

You, Robot.
First, if you doubt it, read On Intelligence.
Second, What is the real value of personal instruction when compared to independent study? It can be high if the teacher and student “click”, but that almost never happens, and it can be negative if the teacher and student don’t get along, which happens more often in schools than it does in the workplace, since the goals of the two places are different.
Third, if objective evidence indicates that teachers can, more often than not, be profitably replaced by “machine-guided learning software”, then why are teachers being paid as much as they are? Could it be that university teachers, like medical doctors, benefit unfairly and unproductively from being in a virtual union?
Fourth, would you say that there is some reason for paying teachers more than I paid for my word processor or library card? Laissez-faire economists want to know.

JWatts May 22, 2012 at 6:50 pm

“In terms of instructor compensation, the researchers estimated, a machine-guided course featuring weekly face-to-face sessions with part-time instructors would cost between 36 and 57 percent less than a traditional course in which a full professor presides over each 40-student section; and it would cost 19 percent less than if a single full professor gave one lecture to all sections before breaking them into smaller discussion groups led by teaching assistants.

The perennial fear among faculty is that the growing credibility of automated teaching software could tempt administrators to replace instructors with robots. But Bowen and company make the case that automated teaching software could enable colleges to save money without firing tenured professors.”

These results, if not a fluke, are pretty astounding. The ramifications are astonishing. Currently the software is:
a) much cheaper,
b) just as effective (at reducing live instruction time from 4 to 1 hours per week) and
c) somewhat duller to the students.

In almost all cases software gets better over time once it reaches a critical user base threshold. Teachers, as a group, generally don’t get any better over time. At least their is scant evidence that Teachers today are better than Teachers from 50 years ago. So if you assume that the software get 1% more effective and 1% more interesting per year and the trend continues over an extended period of time, it seems highly likely that almost all instruction that lends itself to such software would become dominated by it over the next 50 years. I see iTeacher 1.0 in the near future.

cranky critter May 22, 2012 at 8:56 pm

I teach college math and use an online product that automatically administers and grades homework and quizzes. It’s invaluable for providing students with constant and instant feedback, and support when and where they need it: outside class while practicing. I supplement feedback in class. Especially when it comes to broader conceptual understanding.

One thing to notice and look at more is this: success here is being measured by a standardized test that can be autograded. Regardless of subject, not all the things that you need to know and understand can be effectively measured by a standardized test. Such tests are useful tools. So is such software. It can and should be leveraged. This frees instructors to spend more time on critical thinking and applications of concepts, which are more important and interesting, and can’t be assessed by an auto-graded standardized test.

Professors are wasting their time dreading this, both because it’s a useful advance and because it should continue to prove liberating to anyone whose primary concern isn’t job security. But those who think such advances will drastically reduce the need for talented teachers are kidding themselves.

Comments on this entry are closed.

Previous post:

Next post: