Against multiple choice exams

by on October 29, 2008 at 2:50 pm in Education | Permalink

All of the reasons I am opposed to multiple choice exams are predicated on the assumption that I understand the material and that multiple choice exams hinder me from demonstrating what I know. If, however, I didn’t know the material, or didn’t know it well, then I might be led to believe that I’d be better off with a multiple choice exam. After all, with multiple choice exams, at least there would be a chance I could "recognize" something and at least I have something to pick from. On a short answer exam, if I don’t know the material I’ll end up sitting there staring at blank sheets of paper for the entire exam period. Psychologically I think being able to pick an answer to a bunch of questions (even if the answers are wrong) is more comforting than having a bunch a bunch of blank spaces. A 50% is a 50%, regardless of whether you filled in all the bubbles and missed half or if you had a bunch of blank space on a short answer test. But while taking the test, the student will feel more confident filling in the bubbles than anything else.

Here is much more, from this blog, via Andrew Sullivan.  The piece could use a longer discussion of the law of large numbers, but it makes many points of interest.

J.K. October 29, 2008 at 3:36 pm

I always thought it is because of, well, the law of economy.

Multiple choice tests are easy to grade, take less time/energy from the teachers, and thus saving money.

Before we are willing to pay more, much much more, for testing, multiple choice test will be the popular form of testing.

Donald A. Coffin October 29, 2008 at 3:58 pm

Alan Blinder wrote, somehere, long ago, that he didn’t like to give multiple choice tests, because life is not a multiple choice test, it’s an open book essay exam. I agree.

On the other hand, when you have 150 students taking a test, and no grading assistance, then life is a multiple choice exam…

Alex R October 29, 2008 at 4:32 pm

I found the post about multiple-choice exams to be weak, primarily relying on displaying several poorly written multiple-choice questions to discredit the entire concept. Believe me, it’s possible to do a bad job writing essay and short-answer questions as well, though it is easier to do a bad job on multiple-choice questions. He surprisingly didn’t mention what in my experience is the *main reason* that professors use multiple choice exams: not student preference, or perceived objectivity, but simply to reduce the time and effort required for grading. If you are at a school with large lecture classes and few or no TAs to grade exams (or faculty who believe that TAs shouldn’t grade exams), you are going to be getting multiple-choice tests for that reason alone. Good multiple-choice tests do take longer to create than tests with more free-form answers — you usually need more questions to get enough data so that guessing isn’t important, the questions must be carefully written so that there is one clear best answer, and a set of plausible incorrect answers must be created so that one can’t determine the correct answer without having learned what is being tested. The payoff for the extra work comes when you can generate scores for a large class by running the exam sheets through a Scantron. :)

You’re not going to be able to test all types of knowledge with multiple choice tests — though I believe that, with well-written questions, you can do more than you might think. The type of test that you use will depend on what you want the test to be able to do — if you want a performance test for writing, multiple choice is worthless. Testing the ability of students to solve complex problems, giving partial credit for a nearly correct solution, would be pretty challenging to do using multiple choice. But if you want to find out if students have learned a specific body of factual information, they can work very well.

(Oh, and by the way: I thought his Philosophy professor was, if anything, too easy on him in saying that “Martians live on Mars” was a plausible example of a posteriori knowledge — generally, we require something to be *true* for one to be able to know it.)

Anthony October 29, 2008 at 5:22 pm

CFA Exam is actually moving from 4 answer multiple choice questions to 3. Some of them are “plausible wrong answers” so if you remember a formula incorrectly, and calculate an answer, it will be there. Not smart enough to figure out if this is slick or stupid in measuring topic cognition.

Geof October 29, 2008 at 6:12 pm

My problem with multiple choice exams is that they are generally very easy to game. I can usually eliminate two four choices on a question I don’t know the answer to just based on the reasonableness of the wrong answers. Most teachers aren’t good at coming up with believable wrong answers, and always include obviously wrong answers. This created the structural “process of elimination” advantage for the student. The hardest multiple choice test I ever took was one for a Westinghouse Scholarship. The reason was they had eliminated that structural advantage: Every question could have any number of correct answers, including zero.

Kelvin October 29, 2008 at 6:23 pm

As someone who’s marked exams withing composing them, I’m skeptical about whether more markers (which is what you really get by throwing more money at the problem) is really the way out of the time constraint of marking. It was difficult for someone who did not teach the course to be sure that short answer or essay answers covered the necessary material for a certain grade. I found my marking within-question distribution of marks to be bimodal toward 100% and 0%, as it was easy to identify the really strong or weak answers but much more difficult to sort through the middling results. I believe that the instructor would’ve given a more uniform distribution. On the other hand, I was relatively certain that MC answers were testing for what the instructor had taught the students, as opposed to the other questions, which seemed to test more for what I think should’ve been taught.

JD October 29, 2008 at 6:58 pm

The following shows a better method for constructing multiple choice tests: http://www.brics.dk/~mis/multiple.pdf

A Singular Choice for Multiple Choice

Gudmund S. Frandsen, Michael I. Schwartzbach
SIGCSE Bulletin, Vol 38(4), 2006

How should multiple choice tests be scored and graded, in particular when students are allowed to check several boxes to convey partial knowledge? Many strategies may seem reasonable, but we demonstrate that five self-evident axioms are sufficient to determine completely the correct strategy. We also discuss how to measure robustness of the obtained grades. Our results have practical advantages and also suggest criteria for designing multiple choice questions.

Mark October 29, 2008 at 7:10 pm

First, let’s understand that multiple choice tests test what a student DOES NOT know, while an essay or short answer tests what a student DOES know. I find that they go hand-in-hand.

When constructed effectively, a multiple choice exam should have half the questions with one obviously correct answer, where obvious means that had the student attended class, through osmosis they should get it correct.

The next 15 to 20 have two obviously incorrect answers, though even the close seemingly correct answer is obviously incorrect to the better students.

The next 15 have only one obviously incorrect answer and three seemingly correct answers. And lastly, the remaining 10 to 15 questions have four seemingly correct answers.

This allows for better sorting of students’ abilities. I don’t use just multiple choice questions, but I, too, find close correlation between student performances on multiple choice tests and essay or short answer tests.

glh17 October 29, 2008 at 8:15 pm

I give problems/essays in upper-division course and m-c in principles. I’ve tried a combination of m-c, essay, and problems in principles, but now give all m-c. Why? Reason 1: Recently, my school has started using the ETS exam and since this exam is m-c, I think my students are likely to do better. Reason 2: They are easier to make out and grade. I can spend more time doing something else which generally offers greater reward. Reason 3: I can test over a greater range of material. The time limits on my exams are mostly 50 minutes, sometimes 75 minutes. Not much you can ask in the way of essay within those time constraints, especially over several chapters.
I don’t particular like m-c, but given the material we have to cover in principles and the time allotments, they just seem more practical. I really hate the broad cover of the principles courses. I don’t think students get a lot out of those courses.

ufree October 29, 2008 at 9:15 pm

< (Oh, and by the way: I thought his philosophy professor was, if anything, too easy on him in saying that "Martians live on Mars" was a plausible example of a posteriori knowledge--generally we require something to be *true* for one to be able to know it.)>

I read the question differently. I think it asks: which of these statements is empirically verifiable-“can be known” vs. “is known”?

In which case the fact that Descartes did or did not write the essay attributed to him, does not make this answer either more or less correct than answer B: “Martians live on Mars”;

If the question had asked, “which of these statements is an example of a posteriori knowledge, I’d concede the professor’s choice. As it does read, however, I believe there are two correct answers.

Kath October 29, 2008 at 9:31 pm

I had a high school teacher who gave multiple choice exams sometimes with up to 10 potential answers. The first five were single answers then the rest would be combinations of the first five – it was awful – I would much rather take an essay exam over than any day. When studying for the bar exam we were told to try to shine during the short answer and essay – even if we got the answer wrong if our argument was good we would be awarded more points than a right answer poorly argued. OK maybe that says enough about lawyers!

Andy October 29, 2008 at 10:43 pm

I think MC exams can be written well, although it can take some effort. There seem to be more poorly-written MC question perhaps because instructors have to write very many of them and so don’t spend as much time on each one.

ingres October 30, 2008 at 4:43 am

What I find particularly frustrating is knowing the material that I am being tested on but being unable to articulate my ideas properly. To me multiple choice is a way of showing what I know without having to worry about the non-related details of creating words.

The open-endedness of life can be overwhelming. It is an important skill to be able to break it down into a series of discrete choices with limited options.

Think of all the important life skills students will miss out on if they only ever answer essay questions. :)

Neil Strickland October 30, 2008 at 5:39 am

I run a maths course with nearly 300 students. The students enter their answers on a web site which will accept arbitrary mathematical expressions. The answers are checked automatically by code written in Maple, which can do a lot of symbolic mathematics. Many questions are set up to provide feedback depending on fine details of the student’s answer. This took a lot of work to set up, of course, but now it works automatically with no effort, and it is much better than multiple choice. One could do the same for any kind of test where the answers are mathematical expressions. It would be harder for non-mathematical subjects, of course, but I still think that there is potential for much more intelligent computer based assessment.

Rob Pitingolo October 30, 2008 at 9:21 am

ingres, I agree completely with the first sentence you wrote, but I come to a completely different conclusion. To me, MC exams are highly frustrating because they do not allow me to articulate my ideas the way I want. Rather, they force me to articulate them in the narrow way that the exam author wants them to be articulated.

wintercow20, you have an excellent attitude toward your job and your students. I think if there were more professors like you out there, people like me would me much less cynical about higher education.

Donald A. Coffin October 30, 2008 at 12:44 pm

wintercow20–That’s three times a semester for each of three courses. Fortunately, they aren’t all overwhelmingly big.

meter–TAs? Who has TAs? In 35 years, I have had none. But I haven’t taught at schools with large graduate programs in economics.

infopractical October 30, 2008 at 2:48 pm

Perhaps the entire debate is predicated on nonsense. I’ve never understood why anyone assumes that grades in a classroom are the best feedback loop. It’s not that I think they are never valuable, but long before we debate out multiple choice vs. essay we should debate out kids-in-seats-in-a-box vs. kids-doing-real-things.

improbable October 31, 2008 at 6:11 pm

The most interesting thing in the article was a comment describing some clever multiple-choice exams, in which there were several correct answers, assigned different grades, and several wrong answers, assigned different negative grades. I never had anything like that but can see how it would then be a much more useful evaluation, while remaining easy to mark.

The point about saving professors’ time isn’t just about how much of it there is – grading exams is soul-destroying work, if you ask me, and a very poor use of instructors’ time. It takes a lot of time to decode what’s wrong with each student’s answer, but very little of this is communicated back to them. Better to spend that time explaining what’s wrong with some essays to small groups, or something, and keep the evaluation simple.

I’m coming at this from teaching maths & physics courses, if that matters.

velence May 15, 2009 at 1:43 am

students have a lot of work to do. it can give them more press

tom May 15, 2009 at 1:44 am

you are crazy

Comments on this entry are closed.

Previous post:

Next post: