Does classroom time matter?

by on April 2, 2014 at 6:35 am in Economics, Education | Permalink

Maybe not so much.  There is a new NBER working paper by Theodore J. JoyceSean CrockettDavid A. JaegerOnur Altindag, and Stephen D. O’Connell, the abstract is here:

We test whether students in a hybrid format of introductory microeconomics, which met once per week, performed as well as students in a traditional lecture format of the same class, which met twice per week. We randomized 725 students at a large, urban public university into the two formats, and unlike past studies, had a very high participation rate of 96 percent. Two experienced professors taught one section of each format, and students in both formats had access to the same online materials. We find that students in the traditional format scored 2.3 percentage points more on a 100-point scale on the combined midterm and final. There were no differences between formats in non-cognitive effort (attendance, time spent with online materials) nor in withdrawal from the class. Comparing our experimental estimates of the effect of attendance with non-experimental estimates using only students in the traditional format, we find that the non-experimental were 2.5 times larger, suggesting that the large effects of attending lectures found in the previous literature are likely due to selection bias. Overall our results suggest that hybrid classes may offer a cost effective alternative to traditional lectures while having a small impact on student performance.

I do not see an ungated copy, do any of you find one?

1 Huh April 2, 2014 at 7:29 am

I am astonished at the idea that a drop of over 2% in the average grade is considered ‘not much’.

2 andrew' April 2, 2014 at 7:59 am

If we had been at MOOC for 500 years and someone proposed “University” for its 2% improvement in download…

3 Bill April 2, 2014 at 1:47 pm

This isn’t comparing a class to a MOOC. see my comment below.

4 Andrew' April 2, 2014 at 2:32 pm

Yeah, it is.

5 Bill April 2, 2014 at 4:55 pm

Nah, it isn’t.

6 Bill April 2, 2014 at 5:01 pm

Andrew, please go to page three of the paper listed below, and you will see that both groups had access to the same materials, including online video lecture materials.

7 Rahul April 2, 2014 at 8:44 am

FYI: “Two experienced professors taught one section of each format” actually means the first two authors of this paper were themselves teaching the classes.

This is anti-blinding on steroids. If I don’t want my teaching to positively impact student performance, it sure won’t!

8 Andrew' April 2, 2014 at 12:28 pm

I’m cool with nearly all experimental methods.

9 andrew' April 2, 2014 at 7:49 am

2.5 times larger what? Does it take that many people to write that badly?

10 andrew' April 2, 2014 at 7:54 am

This is your abstract, BTW. Do it for you, not for me. By couching it (poorly) in narrow sub-specialty jargon you have made your target audience, well, basically nobody.

I’m very jealous of economists and their working paper racket. I wonder if this is because it is a reader market.

11 Z April 2, 2014 at 8:19 am

This reminds me, I have a working paper on why strangers should send me money. It’s on my blog….

12 Rahul April 2, 2014 at 8:31 am

Engineering Analogy: Build one single short bridge in favorable terrain without any lateral bracing for its trusses, then test it with a lightly loaded car and then boldly overturn a hundred years of empirical observation by publishing your results with a provocative title “Does lateral bracing matter for bridges?”

PS. And intentionally choose over-designed members for test bridge so that there’s no way bracing should matter in your experiment.

13 Cliff April 2, 2014 at 10:10 am

How is that analogous to this study? If your complaint is that it is only one class, the experiment of course needs to be replicated. But according to the authors it differs fundamentally from previous studies by removing a source of selection bias. This is not physics where the reality is easily calculated with formulas. I see no reason to believe the authors rigged the study so that class time could not possibly matter, as you suggest.

14 Andrew' April 2, 2014 at 12:30 pm

The trick is doing it better with the same resources.

One possibility would be to combine the two sections of the class. Then randomize which unit and which students involve the experimental condition.

15 rayward April 2, 2014 at 8:13 am

This isn’t surprising, not for introductory micro, typically very large classes using the lecture format with little or no student participation. Why would it matter if the student and lecturer are physically present in the same room? I have very much enjoyed on-line college courses, which I’ve mostly accessed through i-tunes university. The courses are, for the most part, traditional college courses taught in traditional classrooms filled with full-time students, the only difference being a camera in the classroom. But I was surprised by the lecture format, with very little student participation. I say surprised because I attended law school, where the convention is the Socratic method (at least it was 35 plus years ago when I attended). Of course, the mission of a law school is very different from the mission of a large public university with mostly undergraduates. If the course is taught by the lecture method, why not present the lecture with videotape. Indeed, the best part of i-tunes is that the courses can be downloaded and watched over and over. Would it make a difference if the professor tapes his lectures and the tapes are made available to the students? As the findings of this study seem to indicate, it wouldn’t make much difference in student performance. The issue, the “economic” issue, is whether it would affect the professor. Would recent developments require the professor to update the lecture so often that taping wouldn’t significantly reduce his time lecturing? Young lawyers who choose a specialty in which the law is constantly changing (tax law, for example) sometimes regret not choosing a specialty in which the law never changes (real property law, for example). Many might make the point that micro, like real property law, hasn’t changed in over a hundred years and, therefore, is perfect for the hybrid method.

16 CD April 2, 2014 at 11:55 am

The underlying problem, as you suggest at the beginning, is that this is horrible pedagogy.

p. 7: “Lectures by professors formed the core of ECO 1001. During lectures, the professors presented microeconomic theory and examples using Power Point slides.”

p. 10: “… our fundamental outcome measure is academic performance on exams and the final course grade. … The midterm and final consisted of 30 and 40 multiple choice questions, respectively.”

p. 11: “the midterm and final exams counted for 35 and 45 percent, respectively. The remaining 20 percent of the
course grade comprise online quizzes.”

Econ still hasn’t gotten the news that very little gets learned this way, something most colleagues in math and science figured out decades ago, At a minimum, you want students working problems and explaining their work – multiple choice tests can be an efficient diagnostic but it’s appalling that a course is graded entirely on that basis.

Regardless of how much time they spent in class, these students were swindled.

17 rayward April 2, 2014 at 12:11 pm

I also went through an LLM (Masters of Law) program in tax. There, the professors used the problem method (with Socratic thrown in for good measure) to teach. I too am amazed that students will pay very high tuition just to listen to lectures. It also explains why so many believe that on-line college courses will be an easy transition from the classroom.

18 Andrew' April 2, 2014 at 12:33 pm

“It also explains why so many believe that on-line college courses will be an easy transition from the classroom.”

The amount of my experience with lecturing that could be instantly replaced by on-line is measured in not hours, semesters or years, but decades.

That said, it would be awesome to free up instructors to do it the Hollywood movie versions you speak of.

19 Anonymous Tech Banker April 2, 2014 at 12:44 pm

“I too am amazed that students will pay very high tuition just to listen to lectures.”

Uh, maybe it’s just me, but I’m paying for the *degree*, not the lectures.

20 Lisa George April 2, 2014 at 8:14 am

To better approximate experimental conditions, all course materials for both sections were available online. The experiment is thus perhaps best viewed as the difference between an online course with one class meeting per week rather than two. Viewed in this light, the small exam effect is not surprising. It is also worth noting that the exams are geared toward solving problems: we might expect students to differ more on essay or interpretive questions more closely related to classroom discussion.

21 Nick April 2, 2014 at 9:09 am

I saw this paper presented at AEFP. It’s a well-put together experiment (blinding issues aside, but how could you possibly avoid those?) but the framing is a little misrepresentative. Neither the “online” nor the “classroom” variants of the class represent what you’d typically think of as being “online” or “classroom” versions. The “classroom” variant includes all the online materials, and the “online” variant is accompanied by the exact same lecture material, just stuffed into half the time.

22 mkt April 4, 2014 at 5:48 am

Yes, those are clearly hybrid courses, not online courses. It is still interesting to observe how a movement along the continuum between classroom and online versions has the effect that many of us have been observing or predicting for years: a lower quality learning environment, and less learning. The notion that purely online courses will displace traditional bricks-and-mortar classes is a mirage; the future of education is hybrid, not online.

23 Joseph Ward April 2, 2014 at 9:12 am

I haven’t read the article, but based on the excerpt that Dr. Cowen provided, I wonder if there are bounds where student performance does change significantly. I also wonder if teacher, student, and institutional expectations for professors to grade within a certain range also factors into this.

24 Z April 2, 2014 at 9:47 am

I seem top recall a university somewhere, I want to say California, that decided to drop on-line course because the failure rate was so high. The assumption was the students were slacking off, but it could also be human instructors are more lenient test graders.

25 Andrew' April 2, 2014 at 12:36 pm

“Slacking off” to do what, is the question.

26 GiT April 2, 2014 at 9:14 pm
27 Sean Crockett April 2, 2014 at 9:43 am
28 David Jaeger April 2, 2014 at 10:00 am
29 Bill April 2, 2014 at 10:38 am

Headline doesn’t match the commentary>

Headline: “Does Classroom Time Matter?”

Is this a comparison of Classroom time versus Online, or non-classroom time….no, it is a comparison of one quantity of in person classroom time with another quantity of in person classroom time.

Then does the comment: “Maybe not so much?” match the headline: “Does Classroom Time Matter?”

30 Claude Emer April 2, 2014 at 2:00 pm

If this is what passes as rigourous statistical analysis and passes the threshold for NBER publication, then a lot of things definitely make sense.

Indeed, US behavioral research studies results skew positive
http://www.nature.com/news/us-behavioural-research-studies-skew-positive-1.13599

31 Andrew' April 2, 2014 at 2:42 pm

I sense some of you guys are coming at this from the wrong angle.

We spend nearly a half trillion dollars a year on higher education, and that is with not enough people being educated.

And these are the studies we have.

Comments on this entry are closed.

Previous post:

Next post: