How good are computers as tutors?

by on October 26, 2013 at 4:25 am in Education, Web/Tech | Permalink

Here is one new report:

…the Wolfram Alpha team is launching a new service for learners, the Wolfram Problem Generator, that turns the “computational knowledge engine” on its head.

The Problem Generator – which is available to all Wolfram Alpha Pro subscribers now – creates random practice questions for students, and Wolfram Alpha then helps them find the answers step-by-step.

Right now, the Generator covers six subjects: arithmetic, number theory, algebra, calculus, linear algebra and statistics.

Here is a 2011 Kurt VanLehn paper (pdf) on human vs. computer systems of tutoring:

This article is a review of experiments comparing the effectiveness of human tutoring, computer tutoring, and no tutoring.  “No tutoring” refers to instruction that teaches the same content without tutoring.  The computer tutoring systems were divided by their granularity of the user interface interaction into answer-based, step-based, and substep-based tutoring systems.  Most intelligent tutoring systems have step-based or substep-based granularities or interaction, whereas most other tutoring systems (often called CAI, CBT, or CAL systems) have answer-based user interfaces.  It is widely believed as the granularity of tutoring decreases, the effectiveness increases.  In particular, when compared to No tutoring, the effect sizes of answer-based tutoring systems, intelligent tutoring systems, and adult human tutors are believed to be d = 0.3, 1.0, and 2.0 respectively.  This review did not confirm these beliefs.  Instead, it found that the effect size of human tutoring was much lower: d = 0.79. Moreover, the effect size of intelligent tutoring systems was 0.76, so they are nearly as effective as human tutoring.

One more specific result found in this paper is simply that human tutors very often fail to take advantage of what are supposed to be the advantages of human tutoring, such as flexibility in deciding how to respond to student problems.

By the way, LaunchPad, the new e-portal for our Modern Principles text, contains an excellent adaptive tutoring system.

dearieme October 26, 2013 at 7:42 am

“human tutors very often fail to take advantage of what are supposed to be the advantages of human tutoring, such as flexibility in deciding how to respond to student problems.” So even if some of the tutors are so stupid as to behave like that, humans still narrowly beat machines. Presumably therefore if one selects one’s human tutor wisely, he should beat the machines by a decent margin.

derek October 26, 2013 at 11:47 am

My aunt, Mom’s sister, worked for years at a grocery store at the checkout. This was when prices had to be entered manually. When the scanning machines came in, she was faster than the scanners and as accurate. All the while conversing with the customers, etc. Of course the majority of folks working the checkout were not accurate or fast, so the scanning machines were an improvement.

Oddly, there still are people who check out my groceries, even with all the machines. Even a self checkout gizmo at a store had a woman watching the whole procedure.

So maybe the real application for these is to assist the tutors to be better at what they do.

gwern October 26, 2013 at 1:08 pm

Wise selection of the best tutors does not, unfortunately, scale.

dearieme October 26, 2013 at 4:04 pm

I never really know what people mean by that sort of portentous statement.

Mark Thorson October 26, 2013 at 7:47 pm

It’s not a sustainable platform for integrating innovative frameworks with traditional paradigms.

ShardPhoenix October 26, 2013 at 10:45 pm

He means that maybe carefully selecting a tutor could work for you, but not everyone can do that (due to both limited supply and the fact that not everyone is as discerning as you) – whereas anyone can use one of the computer systems.

Mark Thorson October 27, 2013 at 1:46 pm

That’s what I said!

Nanoprof October 26, 2013 at 9:41 pm

Use automated tutoring to train better human tutors

hoefner October 26, 2013 at 10:04 am

…tutoring, teaching, instruction — but no mention of students or human learning. That betrays the usual pedagogy mindset — focus on very narrow input, rather than broader system environment and output.

Few outside the educational establishment realize how little of formal instructional practice has been evaluated with standard scientific tools – control groups, random assignment, objective data collection, statistics, and systems analysis. Instead, typical classroom practice is set by casual academic studies, fads, custom, ritual, romantic theories, and politics.

Computer directed learning is vastly superior to human tutors/teachers in most subjects for most people. And ‘no tutoring’ is adequate for most people in what they now actually learn & retain in the American K-12+ “education” system.

Rahul October 26, 2013 at 11:12 am

OTOH, the professional educational establishment oversells the results of these “scientific” studies. Often they are small, somewhat contrived, and of limited validity and have serious external validity issues.

Most importantly, proof is in the pudding: Let’s see the output quality of an entire 4-year degree or high-school curriculum designed on the basis of these often extolled scientific principles. I’m waiting eagerly to interact with an online-only trained degree holder and get an impression of his abilities.

FC October 26, 2013 at 12:57 pm

On the Internet, no one knows where you went to school.

Radford Neal October 26, 2013 at 10:46 am

Missing in this discussion is that one thing students should be learning is how to learn without a tutor – from books alone (or from other non-human resources, not including carefully-crafted tutoring systems that won’t exist for most topics, certainly not just-discovered research findings). If you haven’t learned how to do that, your capacity to do things is severely limited.

Roy October 26, 2013 at 7:47 pm

Depends on the subject… And the availability of decent texts. Sure mathematics can be learned easily from a book, but applied fields are quite different. Even in the very text heavy subjects like traditional engineering I think you need some interactive instruction or an immense amount of experience. And I have seen absolutely no evidence that any laboratory subject can be self taught without immense resources and a huge waste of time.

Radford Neal October 26, 2013 at 8:51 pm

You’re right that some things are much more easily taught with human interaction. But even in fields where there is a lot of that sort of thing, there are still going to be many things that can be learned without tutoring, and those who are able to do so will progress much faster. Ultimately, all actually new things have to be learned without a tutor to help.

jdm October 26, 2013 at 11:51 am

I signed up for a free trial this morning thinking it would be useful practice for my 11 year old. I tried a few problems involving the subtraction of polynomials in the Algebra section and most of the time it incorrectly said my answer was wrong. For example, it generated (v^2 + 3v -1) – (5v^2 + v -3) and I typed in -4v^2 +2v + 2 and it came back with “That’s incorrect”. So I asked it what it thought the answer was and it came back with the same thing I had typed. This happened three or four times. I thought maybe I’m not entering the equation in a format that it understands but it actually shows you what it interprets your answer to be and that corresponded what I typed and what it said the answer is (I also tried entering in different ways like -4*v^2 etc and that made no difference). Very strange and certainly not a plus for the computers.

Mark Thorson October 27, 2013 at 10:31 pm

All of the problems with the website will be fixed by the end of November.

Comments on this entry are closed.

Previous post:

Next post: