Debating Economics

Intelligence Squared has held a series of debates in which they poll ayes and nayes before and after.  How should we expect opinion to change with such debates?  Let’s assume that the debate teams are evenly matched on average (since any debate resolution can be written in either the affirmative or negative this seems a weak assumption).  If so, then we ought to expect a random walk; that is, sometimes the aye team will be stronger and support for their position will grow (aye after – aye before will increase) and sometimes the nay team will be stronger and support for their position will grow.  On average, however, we ought to expect that if it’s 30% aye and 70% nay going in then it ought to be 30% aye and 70% nay going out, again, on average. Another way of saying this is that new information, by definition, should not swing your view systematically one way or the other.

Alas, the data refute this position.  The graph shown below (click to enlarge) looks at the percentage of ayes and nayes among the decided Underdogbefore and after.  The hypothesis says the data should lie around the 45 degree line.  Yet, there is a clear tendency for the minority position to gain adherents  – that is, there is an underdog advantage so positions with less than 50% of the ayes before tend to increase in adherents and positions with greater than 50% ayes tend to lose adherents.  What could explain this?

I see two plausible possibilities.

1) If the side with the larger numbers has weaker adherents they could be more likely to change their mind.

2)  The undecided are key and the undecided are lying.

For case 1, imagine that 10% of each group changes their minds; since 10% of a larger number is more switchers this could generate the data.  The problem with 1 and with the data more generally is that we don’t seem to see a tendency towards 50:50 in the world.  We focus on disputes, of course, but more often we reach some consensus (the moon is not made of blue cheese, voodoo doesn’t work and so forth).

Thus 2 is my best guess.  Note first that the number of “undecided” swing massively in these debates and in every case the number of undecided goes down a lot, itself peculiar if people are rational Bayesians.  A big swing in undecided votes is quite odd for two additional reasons.  First, when Justice Roberts said he’d never really thought about the constitutionality of abortion people were incredulous.  Similarly, could 30% of the audience (in a debate in which Tyler recently participated (pdf)) be truly undecided about whether “it is wrong to pay for sex”?  Second, and even more doubtful, could it be that 30% of the people at the debate were undecided–thus had not heard arguments in let’s say the previous 10 years that converted them one way or the other–but on that very night a majority of the undecided were at last pushed into the decided camp?  I think not, thus I think lying best explains the data.

Some questions for readers.  Can you think of another hypothesis to explain the data?  Can you think of a way of testing competing hypotheses?  And does anyone know of a larger database of debate decisions with ayes, nayes and undecided before and after?

Hat tip to Robin for suggesting that there might be a tendency to 50:50, Bryan and Tyler for discussion and Robin for collecting the data.

Comments

People who are weakly attached to their position, or have no particular grounds for belief one way or the other, end up with the majority for social reasons, and these are the people easiest to sway? (Related to your #1, but from a different direction.) It seems like people who hold an unpopular position are going to either need very good reasons for doing so, enough to cut against social pressure (in which case your debate is less likely to supply arguments able to sway them), or are going to be the kind of cantankerous and contrary people who don't care what others thing (in which case your debate is irrelevant to them anyway).

Actually, Alex, I think that concluding undecideds are lying reflects a remarkably obtuse and/or cynical view of human nature. There may in fact be a significant portion of undecideds in any context who already have a decided opinion one way or another, but choose to lie for a number of reasons, including perhaps concern over what other people might think of them, especially if it turns out the consensus post-debate is that they are "wrong."

But I can think of several other explanations off the top of my head. One: the undecided really does have a considered opinion, which he will remember when he thinks about it, but it is buried away in his memory and he literally can't remember it when the pollster asks him. This strikes me as plausible when the undecided is asked his opinion on a subject which he does not normally think about. It may be decided in his mind, but he does not think about it regularly, so it takes time to recall his stand. When unprepared to give an answer, he declares himself undecided and later "remembers" how he feels. I imagine there are vast swathes of questions even highly intelligent academics who pride themselves on thinking for a living would respond to in similar fashion.

Two: the undecided has a bias, or prejudice about the question, but he realizes that his opinion, while firm, is not based on adequate consideration or evidence. He decides to declare himself undecided and listen to the arguments pro and con, to make sure that when he does decide, he has done so after thinking about and hearing different arguments in a much more systematic fashion. Far from being dishonest, this potential stance is in fact steeped in epistemic honesty. "I think I know how I feel about this, but honestly, I know I haven't formed my opinion through careful, systematic evaluation of the pros and cons. Educate me." Your example of how people feel about paying for sex seems a perfect instance of just such a question: after all, everyone already "knows" it is bad to pay for sex, right? Of course, there will be various levels of conscious or unconscious honesty in these instances. Perhaps the undecided pretends to himself he will listen fairly, but simply listens for arguments to confirm his biases rather than taking on board the full range of arguments in a balanced manner.

Three: there are in fact intelligent people who think deeply about issues who prefer to maintain a stance of epistemic agnosticism about certain issues. Justice Roberts and abortion may in fact be a good example of this (I do not know). Such people reserve judgment on important issues until they can take the time to listen to competing viewpoints, weigh the arguments, and come to a reasoned conclusion. All this takes time, thought, and energy, which most people cannot easily spare from their busy lives. There are vast swathes of issues in my intellectual lumber yard, for instance, that I have filed under "Have not decided; review when opportunity presents itself." This is not lying. This is radical epistemic honesty.

Four: there are people who have considered opinions on a subject but who are open to listening to competing arguments and perhaps even new facts. J.M. Keynes comes to mind. They have thought and decided, but they are open to being persuaded otherwise. Of course, they do not have to declare themselves undecided: even a decided listener can change his mind after a debate. But perhaps there is an element of intellectual challenge and incentive-setting they give the debaters by declaring themselves undecided. "Go ahead, big guy, change my mind." While perhaps less honest than arrogant, such a stance is at its base fundamentally an honest one.

Lastly, anyone who thinks they know all the answers all the time, who thinks decisions they made on issues twenty years ago in graduate school are necessarily correct and require no further thought or consideration, are fools and intellectual charlatans. That is the truly dishonest position in this entire debate.

Your assumption "the debate teams are evenly matched on average since any debate resolution can be written in either the affirmative or negative" seems wrong for this series: There appears to be a tendency to formulate in favour of "ayes", especially for extremer cases.

There are a lot of "after-ayes" over 60%, but only one below 40%

This is somewhat true if your graph shows "ayes" as percentage of total decided votes. If your graph shows "ayes" as percentage of all votes, including undecided, then the effect is even stronger.

Do the debaters see the pre-debate votes? If so, it could be that those debaters who are on the underdog side of the issue try harder to convince the audience, since they know they have more people to win over.

Maybe most people are just credulous, weak-minded, and easily swayed? Andromeda makes a good point: maybe weakly-attached people are most easily persuaded one way or another. There's also the question of whether people will change their minds out of sympathy for the underdog team.

A commenter states:

"Actually, Alex, I think that concluding undecideds are lying reflects a remarkably obtuse and/or cynical view of human nature. There may in fact be a significant portion of undecideds in any context who already have a decided opinion one way or another, but choose to lie for a number of reasons, including perhaps concern over what other people might think of them, especially if it turns out the consensus post-debate is that they are "wrong."

I am not sure that Alex meant to imply ill intent when he used the word "lying" (I could be wrong). I read it as just a factual statement that the best reason for the discrepancy is that folks claim to be undecided when in fact they really lean one way or another albeit maybe not very strongly. They might have a "good" reason for "lying" but from an objective point of view the breakdown in the study comes from folks not properly expressing their starting views.

I agree with both of you. I agree that "lying" is probably the cause of the discrepancy but I also think that the reason folks are "lying" is not malicious.

The best cure for this, in my opinion, is to better phrase the starting question by doing one of the following:

(1) Offer 5 options: Strongly Agree, Weakly Agree, Undecided, Weakly Disagree, Strongly Disagree

(2) Qualify the question with a statement that says "Only answer Undecided if you are completely unsure of your stance. If you lean slightly towards Agreeing or Disagreeing, even if you feel that your view is not well enough informed, you should pick a side rather than stating Undecided."

since Y is just a lagged version of X i'm thinking regression to the mean. if you assume that the debate organizers try to avoid topics like "child-molesting, pro or con" and choose well-balanced subjects then the true prior should be about 50% and observed pre-debate deviations from this are mostly picking up random error. the next time you measure opinion on the same issue the random error comes out differently and this appears to be a "pro-minority effect" when really it's just an artifact of the study design.
since most people have low salience and low information on most issues and panel surveys show low reliability in attitude measurement i think it is reasonable to assume there is a large random component to opinions.

When reading it, my reaction was similar to that of Bob Murphy above.

The very nature of indecision is that you alternate between being decided one way and being decided the other way. In the immediate aftermath of a well presented argument it's natural to lean that way.

Looks like a simple variation on "regression to the mean" to me.

Each side has equally talented "convincers", and so will be equally successful at flipping the other side's weaker, marginal adherents.

That will always help the side with the least support to begin with, because it has fewer marginals to flip, while the side with the most support has more.

Thus, if the initial distribution is 70/30, with equally talented debaters I'd always expect the 70-side to lose support and the 30-side to gain it. Only with an initial 50/50 distribution would I expect no change.

Here's a data point: I have a considered opinion on whether it's wrong to pay for sex, but I'm not sure what my answer would be - pigeon-holing a nuanced view into a simple statement is never easy.

I do not think it should be illegal to pay for sex.

I would not pay cash for sex, unless maybe it were to a close friend who was turned on by being treated like a prostitute.

I would not think ill of somebody for offering/accepting money for sex (or the ill thinking would be barely worth a thought).

The thought of sex involving a monetary exchange feels somewhat unpleasant to me.

So is it "wrong"?

I meant to add, relative to the "regression to the mean", explanation, that I don't see how this is an objection to it...

The problem with 1 and with the data more generally is that we don't seem to see a tendency towards 50:50 in the world. We focus on disputes, of course, but more often we reach some consensus (the moon is not made of blue cheese, voodoo doesn't work and so forth).

... as we aren't talking about "the world" and questions that in time can be factually answered definitively ("is the moon made of cheese?") -- we are talking simply of these debates.

And actually there is a lot of tendency towards 50:50 in the real world, if you look where you are likely to find it. For instance, the two major political parties tend to split 50:50 over time, as the result of endless argument and debate.

As Gabriel Rossman points out, regression to the mean is almost surely part of the story.

Suppose that the "before" position is 90% "For". Then for every 1 person the "against" persuades, the "for" would have to persuade 9, just to break even. If undecideds break 50-50, then the after will be closer to 50-50 than the before. If equal shares of the "decideds" also switch then that also leads to regression to the mean. The "for" has a much tougher job if maintaining the same share is the standard.

The extreme version is perhaps even more obvious: suppose that "before" was 100% to 0%. Then it is mathematically impossible for the "after" to be more extreme but it could be more evenly split.

I would guess that the before-majority tends to reflect a consensus opinion, the arguments in favor of which are probably more commonly encountered in everyday life. Most of the arguments and ideas that are new to the audience would then be coming from the anti-consensus side, so more weakly-decided people would break for the side bringing previously unconsidered information to the table.

Americans culturally prefer the underdog. It's the most basic tenet of our history: the outnumbered prevail against overwhelming odds; the disadvantaged succeed despite the machinations of the privileged and powerful; a poor black child in Chicago can become President of the United States.*

So once we know a given position is the underdog, we tend to throw our support in that direction. Conversely, when we know a given position is NOT the underdog, we tend to walk away from it. We want to be on the winning side, and part of our cultural tradition is the idea that the underdog tends to win. This goes all the way back to David and Goliath, and pervades the largely-Protestant upbringing of native-born Americans. American folklore is massively dominated by the victory of the downtrodden.

* - This is not true of Barack Obama, who did not exactly grow up poor, and only reached Chicago in his twenties. What parallels DO exist to this popular inspirational idea from the 1970s are coincidental.

mobile,

economists can do math. Alex already offered your explanation as case 1.

Ezzie,

do more than a quick glance at the post, you are describing case 1.

Its possible that the prevailing information, or rather the information that most people base there assumptions on is bad, and the minority position is based on higher quality information. Or rather, good information is hard to find so less people have it, where as bad information is readily available and thus more people have it. These debates provide an easy source of good information so the minority position is the prevailing position at the end of the debate.

I see that Student of Economics gets it. He is right- consider the extreme case.

@John: I think you and Alex missed the point. The strength of support does not need to be weaker among the majority. The larger side is likely to have more weaker adherents because it is larger, not because of any psychology about being part of the majority. The larger side might also have more stronger adherents and you could still get the same results.

I would say that it is a mistake to use a line rather than a curve. To my eye, using a curve yields something close to a slope of one for the 45-60% range, and the 1/2 slope for the rest of the region.

If my reading is valid, the results could simply mean that consensus views and insular minority views are consistently undermined by exposure to opposing views. Maybe that is indicative of the circumstances under which people expose themselves to differing views.

If most folks agree with you, why discuss it? If most folks disagree with you, well, people are sheeple, or whatever. If a lot of folks, but not a critical majority, both disagree and agree with you, well, maybe the other side is worth looking into on my own, without having to get dragged into a psychological study.

It might be interesting if there were control debates where the issue was not in the least controversial or was scientific or was purely silly or fanciful.

Isn't it the case that the underdog position has had less exposure, and thus retains more potential to convert people? Also, if 5% of the audience switches regardless of their original opinion, the underdog will win every time, right? Or am I missing something?

Frankly, though, the debates I've seen which were "won" by comparing before-and-after polls were so obviously gamed by people voting contrary to their beliefs in the "before" vote that I don't have any faith in the numbers at all.

I'm going to divide the pre-debate votes into four groups: Y, N, U, and S. Y is unswayable yes. N is unswayable no. U is uninformed (decided and swayable). S is stupid. Let's assume that the U crowd breaks down yes/no proportional to the Y/N, they effectively form their views proportionately to how informed people form their views but not by thinking them through. Let's expect the S segment to be totally random, expected to break 50/50 pre and post debate.

Now, in the U crowd, let's say there is a 50% chance that they would take the opposite view. And for argument, let's say that pre-debate, "yes" in the majority opinion. Then we would expect more yeses to switch no than nos to yes. That yields an expected slight bump on the minority opinion. Nobody had to lie.

I would be very curious to see if there is any research into parsing out the number of true 'undecideds' there are in a group. Obviously there is always some degree of prejudice, but allowing for some degree of inclination, it would be interesting to see what percentage of undecideds are actually open to persuasion by argument

I know we're extrapolating from a hypothetical, but in this case I think the specifics of the debate are integral to the outcome.

All of us have been exposed to popular media's salacious presentations of prostitution through movies, pornography, magazines, and such. For media marketing reasons and plain old sexism, the images of sexy women in skimpy clothes are the images we've been inundated with for a long time when we're asked to think about prostitution. When thinking of legalized prostitution my thoughts certainly focused on the women because theirs were the bodies and testimonies presented in pop culture.

I had never thought about the flip side before, the anonymous millions of johns who pay for sex. By framing the question not as "Is it wrong to SELL sex" but "Is it wrong to BUY sex", the invisibility of johns and the lack of discussion about their potential motivations became startlingly clear in a way I'd never considered before. It stands to reason that there are many more johns than prostitutes and still I had never considered them when giving passing thoughts to prostitution.

I'm hard pressed to think of a debate where a demographic as massive and instrumental to the debate as johns are to prostitution had been left almost entirely unquestioned in popular culture before. Who's ever heard of a "Pimp & Ho & John" party? I found that aspect particularly illuminating and I wonder if the undecideds did too.

Alex once again misses the point, because for some reason GMU economists are afraid of math. People cannot become a "double aye" or a "double nay", so the effect of noise is truncated. Please, please- work through the math. Do a monte carlo simulation. This is the sort of thing first-year grad students do in econometrics.

I think nsp is right, though my interpretation is slightly different.

Assuming the performance and quality of arguments of debaters on the night has no correlation with prior opinion (which intuitively I feel is reasonable), we should see the result given.

If 30% of the audience are swayable, and if they all choose a random new opinion (with a 50-50 split) then more of the majority opinion than of the minority will change. This is simple mean regression through the same mechanism as stockmarket picking - it's not that people who used to be good are more likely to become bad, just that they are more likely to become average.

My guess is that there are psychological reasons for the easier and more substantial growth of minority positions. In fact, I would expect similar psychological mechanisms to produce a bias in favor of "nay" positions, assuming there was random allotment for whether debate propositions were phrased positively or negatively. I think people are more picky about what they affirm than what they deny.

Just a math comment on the "reversion to the mean" idea that was mentioned by several other posters.

If p percent of the people were "ayes" before the debate, pa percent of the people are "ayes" after the debate, the probability of a "aye" switching to a "nay" is d, and the probability of "nay" switching to a "aye" is u, then

pa = p*(1-d)+ (1-p)*u.

It would seem that if the teams are equally skilled then perhaps d=u. Then

pa = p*(1-d) + (1-p)*d = p + d - 2*p*d.

So the graph after the debate would stretch from (0, d) to (1, 1-d) where d is the probability that a person changes their mind.

I'm a college debater from outside the US. In the opening poll the audience votes based on their real world knowledge. At the end, they judge based on the hour of entertainment they have just seen.

Policy positions which are less desirable in the real world, can be much more pursuasive in debate-world. You don't need to be nuanced or subtle. Compromises are unnecessary. The opposition, however well-informed, is unlikely to have a response to the particular peice of evidence you selectively present. Judges will often vote for the most outlandish policies that would never gain support in the real world if they are packaged well enough in that hour.

I think people aren't voting on what they actually believe to be true in the real world, but who entertained them, or had the most chutzpah, or whatever - even in the IQ2 debates which I listen to. Comparing the two polls is essentially meaningless because they are measuring different things. And a less popular position in the real world of subtlety and compromise can be much more pursuasive for the hour that the audience is living in debate-world.

Most people (the great majority) have uninformed opinions, usually formed due to an emotional appeal. As such, the majority opinion is most fragile.

This is why I always say "the majority is usually wrong" and why opinion polls are not only meaningless, but actually dangerous. I assume the opposite of whatever an opinion poll shows.

It's funny you attack the undecideds. I actually came to conclude this proposition is undecidable, as formulated, during the debate.

MacKinnon basically argued that paying for sex leads to a great deal of abuse of women, and this latter thing makes the former wrong. Cowen pointed out that abuse of women and paying for sex are not intrinsically connected, even if frequently. They are talking about two different propositions.

If we say "It is wrong to murder," then presumably we mean the act itself is inherently wrongful. Much different if we say "It is wrong to drive on the left side of the road." Nothing inherently bad about driving there, it just tends to lead to great harm. These are different senses of the phrase "It is wrong to x", and we don't know which one is to be debated.

If "It is wrong to pay for sex" means "The wrongfulness of paying for sex is in the paying," then I think Cowen clearly wins. If "It is wrong to pay for sex" means "Paying for sex tends to lead to other serious wrongfulness", then I think MacKinnon might have won that debate (NZ aside).

I found the proposition undecidably vague as formulated.

Thanks for your useful info, I think it’s a good topic.

Excellent site. It was pleasant to me.

Comments for this post are closed