Category: Education

ASA Against Student Evaluations

Yesterday in Active Learning Works But Students Don’t Like It I pointed out that student evaluations do not correlate well with teacher effectiveness and may discourage teachers from using more effective but less student-preferred methods of teaching. Coincidentally the American Sociological Association issued a statement yesterday discouraging student evaluations for tenure and promotion decisions.

SETs are weakly related to other measures of teaching effectiveness and student learning (Boring, Ottoboni, and Stark 2016; Uttl, White, and Gonzalez 2017); they are used in statistically problematic ways (e.g., categorical measuresare treated as interval, response rates are ignored, small differences are given undue weight, and distributionsare not reported) (Boysen 2015; Stark and Freishtat 2014); and they can be influenced by course characteristics like time of day, subject, class size, and whether the course is required, all of which are unrelated to teaching effectiveness. In addition, in both observational studies and experiments, SETs have been found to be biased against women and people of color (for recent reviews of the literature, see Basow and Martin 2012 and Spooren, Brockx, and Mortelmans 2015).

Student evaluations mostly evaluate entertainment value but, as my colleague Bryan Caplan notes, given how boring most classes are, entertainment value is worth something! Thus, the ASA notes that student evaluations can be useful as a form of feedback.

Questions on SETs should focus on student experiences, and the instruments should be framed as an opportunity for student feedback, rather than an opportunity for formal ratings of teaching effectiveness.

Student evaluations are on their way out in Canada, as a tool for tenure and promotion. I expect the same here, at least on paper. The ASA is big on “holistic” measures of teacher evaluation as a replacement which strikes me as weaselly. I’d prefer more objective measures such as value added scores.

Active Learning Works But Students Don’t Like It

A carefully done study that held students and teachers constant shows that students learn more in active learning classes but they dislike this style of class and think they learn less. It’s no big surprise–active learning is hard and makes the students feel stupid. It’s much easier to sit back and be entertained by a great lecturer who makes everything seem simple.

Despite active learning being recognized as a superior method of instruction in the classroom, a major recent survey found that most college STEM instructors still choose traditional teaching methods. This article addresses the long-standing question of why students and faculty remain resistant to active learning. Comparing passive lectures with active learning using a randomized experimental approach and identical course materials, we find that students in the active classroom learn more, but they feel like they learn less. We show that this negative correlation is caused in part by the increased cognitive effort required during active learning. Faculty who adopt active learning are encouraged to intervene and address this misperception, and we describe a successful example of such an intervention.

The authors say that it can help to tell students in advance that they should expect to feel flustered but it will all work out in the end.

The success of active learning will be greatly enhanced if students accept that it leads to deeper learning—and acknowledge that it may sometimes feel like exactly the opposite is true.

I am dubious that this will bring students around. An alternative that might help is to discount student evaluations so that teachers don’t feel that they must entertain in order to do well on evaluations. As Brennan and Magness point out in their excellent Cracks in the Ivory Tower:

Using student evaluations to hire, promote, tenure, or determine raises for faculty is roughly on a par with reading entrails or tea leaves to make such decisions. (Actually, reading tea leaves would be better; it’s equally bullshit but faster and cheaper.)… the most comprehensive research shows that whatever student evaluations (SETs) measure, it isn’t learning caused by the professor.

Indeed, the correlation between student evaluations and student learning is at best close to zero and at worst negative. Student evaluations measure how well liked the teacher is. Students like to be entertained. Thus, to the extent that they rely on student evaluations, universities are incentivizing teachers to teach in ways that the students like rather than in ways that promote learning.

It’s remarkable that student evaluations haven’t already been lawsuited into oblivion given that student evaluations are both useless and biased.

The new generational divide

That is the topic of my latest Bloomberg column, here is one bit:

In a nutshell, younger people today are very comfortable with a small screen and older people are not. Both younger and older people can be found staring at their phones for texts or email or directions, but the big difference comes in cultural consumption. According to one study, the median age of an American television viewer is about 56, whereas for mobile and computer video viewers the median age is 40. Forty percent of those viewers are between 13 and 34…

Just as many older people don’t grasp the import of YouTube, most younger people have a weak sense of the power of cinema on a large screen. It’s not entirely their fault. It’s relatively easy to see older movies on a big screen in London or Paris, and maybe in New York City and Los Angeles (and Silver Spring, Maryland, home to the American Film Institute). In most other places in America, it’s much more difficult.

Sadly, the world is rapidly becoming a place where cinematic history, as it was created for larger screens, no longer exists. Netflix, for all its wonders and diverse contemporary selection, is notoriously bad about making older movies available for streaming, and at any rate the service does not provide a properly large screen for those films.

There is much more at the link, and the economically-minded reader will note this is an application of the Alchian-Allen Theorem.

Ideological bias and argument from authority among economists

That is the topic of a new paper by Mohsen Javdani and Ha-Joon Chang, here is part of the abstract:

Using an online randomized controlled experiment involving economists in 19 countries, we examine the effect of ideological bias on views among economists. Participants were asked to evaluate statements from prominent economists on different topics, while source attribution for each statement was randomized without participants’ knowledge. For each statement, participants either received a mainstream source, an ideologically different less-/non-mainstream source, or no source. We find that changing source attributions from mainstream to less-/non-mainstream, or removing them, significantly reduces economists’ reported agreement with statements. This contradicts the image economists have of themselves…

And from the paper:

Consistent with our overall findings, we find that for all but three statements, changing source attributions to a less/non-mainstream source significantly reduces the agreement level. The estimated reductions range from around one-tenth of a standard deviation to around half of a standard deviation.

The largest agreement reduction is for this sentence:

“Economic discourse of any sort — verbal, mathematical, econometric — is rhetoric; that is, an effort to persuade.”

You also can test which kinds of authority reassignation alter the level of agreement.  And thus:

We find that the estimated ideological bias among female economists is around 40 percent less than their male counterparts.

The countries where economists exhibit the highest ideological bias are Ireland, Japan, Australia, and Scandinavia, where for Austria, Brazil, and Italy the ideological bias is smallest.  South Africa, France, and Italy are most conformist to mainstream opinion.

It is a wordy and poorly written paper, and they don’t consider the possibility that deference to authority perhaps is the rational Bayesian move, not the contrary.  Still, it has numerous results of interest.  Here is the authors’ blog post on the paper.

Emergent Ventures winners, fifth cohort

James Gallagher

16-year-old programmer from Northern England, grant for career development and his interest in income-sharing agreements.  Here is James on Twitter.

Namrata Narain

Incoming Harvard Ph.D student in economics, for work on “What happens to the ability of firms to write contracts when courts are dysfunctional? [in India]” and related ideas.  Twitter here.

Tejas Subramaniam

17-year-old from Chennai, Twitter here.  

Andy Matuschak

San Francisco, to support his project to reexamine and fundamentally improve the book as a method for learning and absorbing ideas, Twitter here.  Here is his essay on why books do not work.

Nicholas Donahue and Austin Kahn

Has a start-up, open source VR headset focused towards makers and web developers, based on the notion that the web is the proper platform for VR.

Clementine Jacoby and Recidiviz

To start a non-profit to collect and spread data on recidivism and penal reform for state-level policy, Fast Company article on Recidiviz here.

Mehdi Nayebpour

GMU, Schar School, “How can we explain a specific AI outcome? What if the law mandates it?”, with an eye toward an eventual start-up.

Colin Mortimer

Washington, D.C., for career development and to explore the marketing of neoliberal ideas through social media.

Shruti Rajagopalan, for Indian political economy and improving Indian economic policy, in residence at Mercatus.  Twitter here.

Jasmine Wang

20-year-old infovore, career development grant, Twitter here.


A non-profit working with survivors of sexual abuse in the Catholic Church, article here about their work.

If you have received an award lately, but are not listed, don’t worry — you’ll be in the sixth cohort.  Here are the earlier cohorts of winners.

Is this the very best book ever written?

No, I don’t mean Proust, Cervantes, or the Bible.  I mean Dave Marsh’s The Heart of Rock and Soul: The 1001 Greatest Singles Ever Made.

To be sure, it is not the greatest book qua book, or even in the top tier (though it is very good and Marsh is very smart and knowledgeable).

It is possible it has become the greatest book of all time because of YouTube.  Scroll through the pithy, one-page or sometimes even one-paragraph reviews of the various songs, and play them on YouTube while you are reading.

I had not known of Marvin Gaye’s “One More Heartache,” or Aretha Franklin’s “Think.”  Nor had I known the live version of Bob Dylan’s “Just Like Tom Thumb’s Blues” from 1966 (though is it really “Slurred and obtuse as Little Richard reading Ezra Pound”?).  I heard again many favorites as well.

Let’s be honest, amusia aside, do not humans love music more than books?  By no means does everyone read, but virtually everyone listens to music, and with some degree of passion. It therefore follows that “book + music” is better than book, right?  Whatever virtues the book may have are still contained in “book + music,” or more generally “book + YouTube.”

Have we now entered an age where all or most of the very best books are part of “books + YouTube”?

Of course I’m not trying to sell you on music or for that matter on Dave Marsh.  What about reading Abraham Pais, Niels Bohr’s Times: In Physics, Philosophy, and Polity, accompanied by these videos?  Might the possibility of YouTube combination make that the 37th best book of all time, displacing Braudel or Flaubert?

Should not at least 2/3 of your reading be books accompanied by YouTube?  And if not, why not?

Inquiring minds wish to know.  Perhaps there is a book accompanied by YouTube that gives the answer?

Is a quality book better or worse if there is no useful way to combine it with YouTube?

Addendum: You will note that the Cowen-Tabarrok Modern Principles text can be combined with our micro and macro videos on YouTube, and thus it is one of the best books, not just our favorite.

Girls’ comparative advantage in reading can largely explain the gender gap in math-related fields

In an earlier post, Do Boys Have a Comparative Advantage in Math and Science? I pointed to evidence showing that boys have a comparative advantage in math because they are much worse than girls at reading. (Boys do not have a large absolute advantage in math.) If people specialize in their personal comparative advantage this can easily lead to more boys than girls entering math training even if girls are equally or more talented. As I wrote earlier:

[C]onsider what happens when students are told: Do what you are good at! Loosely speaking the situation will be something like this: females will say I got As in history and English and B’s in Science and Math, therefore, I should follow my strengthens and specialize in drawing on the same skills as history and English. Boys will say I got B’s in Science and Math and C’s in history and English, therefore, I should follow my strengths and do something involving Science and Math.

A new paper in PNAS by Breda and Napp finds more evidence for the comparative advantage hypothesis. Breda and Napp look at intention to study math in ~300,000 students worldwide taking the PISA.

PISA2012 includes questions related to intentions to pursue math-intensive studies and careers. These intentions are measured through a series of five questions that ask students if they are willing (i) to study harder in math versus English/reading courses, (ii) to take additional math versus English/reading courses after school finishes, (iii) to take a math major versus a science major in college, (iv) to take a maximum number of math versus science classes, and (v) to pursue a career that involves math versus science. Our main measure of math intentions is an index constructed from these five questions and available for more than 300,000 students. It captures the desire to do math versus both reading and other sciences.

What they find is that comparative advantage (math ability relative to reading ability) explains math intentions better than actual math or reading ability. Comparative advantage is also a better predictor of math intentions than perceptions of math ability (women do perceive lower math ability relative to true ability than do men but the effect is less important than comparative advantage). In another data set the authors show that math intentions predict math education.

Thus, accumulating evidence shows that over-representation of males in STEM fields is perhaps better framed as under-representation of males in reading fields and the latter is driven by relatively low reading achievement among males.

As the gender gap in reading performance is much larger than that in math performance, policymakers may want to focus primarily on the reduction of the former. Systematic tutoring for low reading achievers, who are predominantly males, would be a way, for example, to improve boys’ performance in reading. A limitation of this approach, however, is that it will lower the gender gap in math-intensive fields mostly by pushing more boys in humanities, hence reducing the share of students choosing math.

The authors don’t put it quite so bluntly but another approach is to stop telling people to do what they are good at and instead tell them to do what pays! STEM fields pay more than the humanities so if people were to follow this advice, more women would enter STEM fields. I believe that education spillovers are largest in the STEM fields so this would also benefit society. It is less clear whether it would benefit the women.

Hat tip: Mary Clare Peate.

My Conversation with Hollis Robbins

Here is the audio and video, here is part of the CWT summary:

Now a dean at Sonoma State University, Robbins joined Tyler to discuss 19th-century life and literature and more, including why the 1840s were a turning point in US history, Harriet Beecher Stowe’s Calvinism, whether 12 Years a Slave and Django Unchained are appropriate portraits of slavery, the best argument for reparations, how prepaid postage changed America, the second best Herman Melville book, why Ayn Rand and Margaret Mitchell are ignored by English departments, growing up the daughter of a tech entrepreneur, and why teachers should be like quarterbacks.

Here is one excerpt:

COWEN: You’ve written a good deal on the history of the postal service. How did the growth of the postal service change romance in America?

ROBBINS: Well, everybody could write a letter. [laughs] In 1844 — this was the other exciting thing that happened in the 1840s. Rowland Hill in England changed the postal service by inventing the idea of prepaid postage. Anybody could buy a stamp, and then you’d put the stamp on the letter and send the letter.

Prior to that, you had to go to the post office. You had to engage with the clerk. After the 1840s and after prepaid postage, you could just get your stamps, and anybody could send a letter. In fact, Frederick Douglass loved the idea of prepaid post for the ability for the enslaved to write and send letters. After that, people wrote letters to each other, letters home, letters to their lovers, letters to —

COWEN: When should you send a sealed letter? Because it’s also drawing attention to itself, right?

ROBBINS: Well, envelopes — it’s interesting that envelopes, sealed envelopes, came about 50 years after the post office became popular, so you didn’t really have self-sealing envelopes until the end of the 19th century.

COWEN: That was technology? Or people didn’t see the need for it?

ROBBINS: Technology, the idea of folding the envelope and then having it be gummed and self-sealing. There were a number of patents, but they kept breaking down. But technology finally resolved it at the end of the 19th century.

Prior to that, you would write in code. Also, paper was expensive, so you often wrote across the page horizontally and then turned it to the side and crossed the page, writing in the other direction. If somebody was really going to snoop on your letters, they had to work for it.

COWEN: On net, what were the social effects of the postal service?

ROBBINS: Well, communication. The post office and the need for the post office is in our Constitution.

COWEN: It was egalitarian? It was winner take all? It liberated women? It helped slaves? Or what?

ROBBINS: All those things.

COWEN: All those things.

ROBBINS: But yeah, de Tocqueville mentioned this in his great book in the 1830s that anybody — some farmer in Michigan — could be as informed as somebody in New York City.


COWEN: Margaret Mitchell or Ayn Rand?

ROBBINS: Well, it’s interesting that two of the best-selling novelists of the 20th-century women are both equally ignored by English departments in universities. Margaret Mitchell and Gone with the Wind is paid attention to a little bit just because, as I said, it’s something that literature and film worked against, but not Ayn Rand at all.


COWEN: What’s a paradigmatic example of a movie made better by a good soundtrack?

ROBBINS: The Pink Panther — Henry Mancini’s score. The movie is ridiculous, but Henry Mancini’s score — you’re going to be humming it now the rest of the day.


COWEN: What is the Straussian reading of Babar the Elephant?

ROBBINS: When’s the last time you read it?

COWEN: Not long ago.

Recommended throughout.

Watch out for the weakies!: the O-Ring model in scientific research

Team impact is predicted more by the lower-citation rather than the higher-citation team members, typically centering near the harmonic average of the individual citation indices. Consistent with this finding, teams tend to assemble among individuals with similar citation impact in all fields of science and patenting. In assessing individuals, our index, which accounts for each coauthor, is shown to have substantial advantages over existing measures. First, it more accurately predicts out-of-sample paper and patent outcomes. Second, it more accurately characterizes which scholars are elected to the National Academy of Sciences. Overall, the methodology uncovers universal regularities that inform team organization while also providing a tool for individual evaluation in the team production era.

That is part of the abstract of a new paper by Mohammad Ahmadpoor and Benjamin F. Jones.

Shruti Rajagopalan update

Next week Dr. Shruti Rajagopalan (also an Emergent Ventures winner) will be joining Mercatus as a senior research fellow, focused on Indian political economy, property rights, and economic development.

Shruti earned her PhD from George Mason in 2013 and most recently she is an associate professor of economics at Purchase College, SUNY. She is also a fellow of the Classical Liberal Institute at NYU, and will be teaching at the newly-formed Indian School of Public Policy in New Delhi.

Welcome Shruti!

You can follow Shruti on Twitter here.

My Conversation with Masha Gessen

Here is the transcript and audio, here is the summary:

Masha joined Tyler in New York City to answer his many questions about Russia: why was Soviet mathematics so good? What was it like meeting with Putin? Why are Russian friendships so intense? Are Russian women as strong as the stereotype suggests — and why do they all have the same few names? Is Russia more hostile to LGBT rights than other autocracies? Why did Garry Kasparov fail to make a dent in Russian politics? What did The Americans get right that Chernobyl missed? And what’s a good place to eat Russian food in Manhattan?

Here is excerpt:

COWEN: Why has Russia basically never been a free country?

GESSEN: Most countries have a history of never having been free countries until they become free countries.


COWEN: But Russia has been next to some semifree countries. It’s a European nation, right? It’s been a part of European intellectual life for many centuries, and yet, with the possible exception of parts of the ’90s, it seems it’s never come very close to being an ongoing democracy with some version of free speech. Why isn’t it like, say, Sweden?

GESSEN: [laughs] Why isn’t Russia like . . . I tend to read Russian history a little bit differently in the sense that I don’t think it’s a continuous history of unfreedom. I think that Russia was like a lot of other countries, a lot of empires, in being a tyranny up until the early 20th century. Then Russia had something that no other country has had, which is the longest totalitarian experiment in history. That’s a 20th-century phenomenon that has a very specific set of conditions.

I don’t read Russian history as this history of Russians always want a strong hand, which is a very traditional way of looking at it. I think that Russia, at breaking points when it could have developed a democracy or a semidemocracy, actually started this totalitarian experiment. And what we’re looking at now is the aftermath of the totalitarian experiment.


GESSEN: …I thought Americans were absurd. They will say hello to you in the street for no reason. Yeah, I found them very unreasonably friendly.

I think that there’s a kind of grumpy and dark culture in Russia. Russians certainly have a lot of discernment in the fine shades of misery. If you ask a Russian how they are, they will not cheerfully respond by saying they’re great. If they’re miserable, they might actually share that with you in some detail.

There’s no shame in being miserable in Russia. There’s, in fact, a lot of validation. Read a Russian novel. You’ll find it all in there. We really are connoisseurs of depression.

Finally there was the segment starting with this:

COWEN: I have so many questions about Russia proper. Let me start with one. Why is it that Russians seem to purge their own friends so often? The standing joke being the Russian word for “friend” is “future enemy.” There’s a sense of loyalty cycles, where you have to reach a certain bar of being loyal or otherwise you’re purged.

Highly recommended.