Is America as polarized as you think?

by on March 13, 2018 at 2:01 am in Current Affairs, Data Source, Political Science | Permalink

In this study, we argue that the perceived polarization of Americans along party lines is partially an artifact of the low response rates that characterize contemporary surveys. People who agree to participate in opinion surveys are more informed, involved, and opinionated about the political process and therefore hold stronger, more meaningful, and partisan political attitudes. This motivational discrepancy generates a bias in survey research that may amplify evidence of party polarization in the mass public. We test the association between response rates and measures of polarization using individual-level data from Pew surveys from 2004 to 2014 and American National Election Studies from 1984 to 2012. Our empirical evidence demonstrates a significant decline in unit response that is associated with an increase in the percentage of politically active, partisan, and polarized individuals in these surveys. This produces evidence of dissensus that, on some issues, may be stronger than exists in reality.

That is from a forthcoming piece by Cavari and Freedman in The Journal of Politics.  For the pointer I thank an anonymous correspondent.

And via Nathan, here is a relevant comment.

1 Chris Prosser March 13, 2018 at 5:30 am

It’s an intuitive argument and there might be something to it but the evidence presented in the paper is likely spurious because both polarization and response rates are trending. My co-author Jon Mellon and I re-examined Cavari and Freedman’s data and find that when you control for time to account for the trending nature of the data there is no longer a statistical relationship between response rate and polarization. We also demonstrate the likelihood that the result is spurious by showing you get almost identical results when you replace response rate with another trending variable that obviously has no connection with polarization – per capita milk consumption. Paper here:


2 mavery March 13, 2018 at 8:44 am

Can you elaborate? Or provide an ungated link? I’m unable to access the paper via the link you provided and would like to read it. In the meantime…

It sounds like you’re making an argument analogous to the classic example of positive correlation between murder rates and ice cream consumption. In the murder/ice cream example, there is a positive correlation between the two variables because both are correlated with temperature. All of these have similar trends over a fixed time period. People are out and about more in warmer times, so they interact more, so there’s more murder, or so went the argument in my Econ 101 course 15 years ago. (The correlation between temperature and ice cream consumption should be obvious.) So you have a spurious correlation due to an ignored third variable, all three of which are correlated temporally.

In the post you mentioned above, however, you don’t posit what the ignored variable is. You merely show that three things that show trends over the same time period are correlated with each other. While this does make the arguments in the paper you’re responding to less convincing, you go to far in stating that the result is spurious. Returning to the above example, milk consumption could be the ice cream and murder rate the response rate, while there truly is a causal relationship between response rate and polarization. (Since there’s no clear relationship between polarization and milk consumption as far as I can tell, the analogy is imperfect.)

If we discarded all correlations between variables where we have a theoretically justified mechanism for the correlation, we’d get nowhere fast. Finding a third variables that correlated with the other two doesn’t prove a claim spurious. It just buys you an argument about which theoretical mechanism is more palatable or a search for a better data set. From your abstract and summary here, I don’t find your argument very persuasive.

As I said above, I would like to read the full paper if you have an ungated version. There may be more convincing arguments therein.


3 Peter March 13, 2018 at 9:28 am

I can’t speak to the above, but my immediate thought when reading the original post is that it’s exactly what you’d expect, because the only people who DO consistently vote and argue about politics are becoming more polarized. Response rates would go down for the increasing majority who just don’t want to get caught up in all that. So perhaps increasing polarization in a subset of the population simply reduces the number of people in the middle who are willing to even respond to a survey about politics.


4 Chris Prosser March 13, 2018 at 9:57 am

You should be able to download the paper from the above link – SSRN is an open access depository – but sometimes it gets funny when lots of people go to the same paper in a short space of time and tries to make you sign in. Try this link instead:

The ice-cream/murder is basically the same point we’re making but you don’t actually need to know that temperature is the omitted variable to know that the relationship is spurious – you can adjust for the seasonality of both variables and find that the correlation between them is due to that seasonality without necessarily knowing why both are seasonal.

Likewise we don’t need to know what the omitted variables are between milk consumption, polarization, and time to know that is clearly spurious – you just control for the fact that both are trending. The problem of spurious correlations due to time trends is very well known in statistics – see here for example,%20ABA%20newsletter%202015.pdf. You can have fun exploring clearly spurious relationships that are due to trending variables here

Controlling for the trend doesn’t get rid of any correlation that isn’t due to the trend, so if there was a relationship between response rate and polarization that wasn’t just an artifact of the time correlation you would still expect to see it in a regression model controlling for the time trend, and we don’t.

The milk consumption example may seem a bit flippant but the point is very serious – the statistical evidence for a relationship between milk consumption is actually stronger than the statistical relationship between response rate and polarization. That statistical evidence is obviously misleading but it looks exactly like the relationship between response rate and polarization. How can we know that one is spurious but the other isn’t? Without a properly identified model, we can’t. Our point is not that there is no relationship between response rate and polarization, and we are not saying we have a properly identified model that shows no relationship. Our point is that this analysis claims to have evidence of that relationship but when examined properly does not.


5 Curt F. March 13, 2018 at 10:22 am

I don’t understand this at all. Monotonic correlation with time doesn’t seem to imply the presence of an unknown confounding variable. It seems like saying, “I spent 5 seconds slowly turning up the volume knob on my TV, and sure, the sound output increased, but you can’t conclude that the volume knob is related to the sound output, because if you control for time, the effect of the volume knob setting disappears.” In contrast, the seasonality adjustment makes much more sense to me. Can you explain why non-periodic effects for “time” in a regression necessarily indicate a missing variable?


6 Chris Prosser March 13, 2018 at 11:25 am

To stretch your TV example a bit (perhaps too far), imagine two situations in which you can measure time, the position of the volume control, and the volume level perfectly:
1) You increase the volume of your TV at random intervals.
2) You press the volume control but it is broken and your TV is playing a recording of an increasingly loud noise.

Both situations are correlated with time and the position of the volume control but in the first case the time trend is spurious and in the second case the control trend is spurious. A statistical model of TV volume as a function of control position and time would be able to distinguish these two situations because conditional on the other variable there is no correlation between volume and spurious variable.

The only exception to this is the situation in which you adjust the volume control at perfectly regular intervals, in which case control position and time are perfectly correlated and so a model with them both would be impossible to estimate. Perfect correlations don’t generally happen in reality so this is rarely a problem for actual trending variables. In any case if you were actually in this situation you wouldn’t be able to tell if you were in scenario 1 or 2, in which case you can either take it on faith that your volume control works or come up with a better test.

7 mavery March 13, 2018 at 11:31 am

“Controlling for the trend doesn’t get rid of any correlation that isn’t due to the trend, so if there was a relationship between response rate and polarization that wasn’t just an artifact of the time correlation you would still expect to see it in a regression model controlling for the time trend.”

This is only true if the time trend is the “real” trend. If your data generation process is:
x = f(t)
y = f(x)

(where t is time), then x and y will be time-correlated. If the time correlation with X is particularly strong, you will see the phenomena you describe, where after accounting for the time dependency, x and y have little remaining correlation. But that’s not because y and x are independent.

Perhaps a better way to say it is that x|t and y|t can appear independent if x and y are both highly correlated with t, even if there is in fact a direct linear relationship between x and y. If you want an example and have R, see below.

t <- 0:100
x <- 40 + 2 * t + rnorm(length(t))
y <- .1 * x + rnorm(length(x), sd = .15)
xr % summary
lm(y ~ t + x) %>% summary
lm(y ~ xr) %>% summary

“That statistical evidence is obviously misleading but it looks exactly like the relationship between response rate and polarization. How can we know that one is spurious but the other isn’t?”

This one is simple. You believe the first model because you have a clearly-identified, pre-specified mechanism that you’re testing when it comes to response rate and polarization, whereas you have neither of these when you compare milk consumption and response rate. If you’re trying to make a “garden of forking paths” argument, you have to engage the process by which the original authors arrived at their model. In this case, nonresponse bias is an obvious phenomena to consider when looking at survey data. In fact, it would be surprising (IMO) if changing response rates over time had no effect on the characteristics of the sample.

“Our point is not that there is no relationship between response rate and polarization, and we are not saying we have a properly identified model that shows no relationship. Our point is that this analysis claims to have evidence of that relationship but when examined properly does not.”

I think I actually generally agree with you about this. I only think that you’re overstating your case. I don’t think its prudent to throw out every analysis just because the data set they’re looking at only has a temporally monotonic trend, but I do think its fair to say that drawing any causal inference at all is challenging given the structure of their data.


8 Chris Prosser March 13, 2018 at 12:07 pm

When I run your code I get a coefficient for x of 0.096 with a p value of 2.66e-09 and a coefficient for t of 0.006 with a p value of 0.827, or in other words, the model closely approximates the data generating process despite the (spurious) correlation with t, which is what I would expect given what I said above. But you seem to be saying that we would wrongly conclude they were independent in this example, which is not what the code shows, so I don’t follow.

In any case to address your substantive point about theory. We’re not approaching this as a statistical ‘gotcha’ moment without any grounding in theory. We are survey researchers and think that non-response bias is a very important threat to valid inference in survey research and have published saying exactly that (see here for example The non-statistical problem with Cavari and Freedman’s analysis is that they equate response *rate* with response *bias*. As we discuss in our response paper, there is loads of research that shows that response rate and bias are not the same thing, and that you can have equally unbiased surveys with very different response rates. There is no good evidence that response bias has increased over time even though response rates have fallen.

9 mavery March 13, 2018 at 3:44 pm

First off, sorry for the hideous formatting. It’s a minor miracle you took the time to untangle that.

My point with the example (which I’ll grant wasn’t clear at all) was that it depends on how you do the analysis. If you include both terms, you could get a response that is significant, as in the example above. (Note that this can disappear as the correlation between x and t increases; replace the existing line with x <- 40 + 2 * t + rnorm(length(t), sd = .1) and the significance of x|t disappears.) If you attempt to "detrend" the data and do your regression on the residuals (this was the last example), you find that there's no relationship between y and x.

As for the other part (the "non-statistical problem" as you put it), I believe it can be attributed to some sloppy language on the part of Cavari and Freedman. Although you interpret their paper to be equating changes in response rate to response bias, my reading is that this is their research question. From the paper:

"As the response rate decreased in recent years—and especially when it reached the extremely low levels that exist today—surveys include a larger share of an engaged and partisan public, who are polarized along party lines. Because of this bias, surveys project a more polarized image of the American public than exists in reality."

Here, "bias" specifically refers to the singular question of polarization, and they spend the rest of the paper trying to justify this statement. They're not presupposing nonresponse bias so much as trying to show it. In fact, they seem pretty willing to admit that other measures might be fine even with the low response rates:

"There is no doubt that despite low response rates, a well-done probability sample can allow researchers to generalize about large populations. Yet, survey data may not adequately represent the attitudes of all Americans if the bias in samples is correlated with the topic of interest. This may be the case regarding measures of polarization in contemporary surveys."

To me, this reads as pretty circumspect.

Anyhow, I'll stop here. I'm not an expert on public policy polling, (I even managed to avoid taking Sampling in grad school) so there may be some subtleties that I'm missing. But in my mind at least, showing that there are other covariates that are similarly monotonic over the same timer period is best framed as a, "Well, not necessarily…" type of finding.

PS- I'm still interested in reading the papers you mentioned if there's a way to get at them ungated.

10 Albert Rio March 13, 2018 at 6:07 am

“Our results also showed that Facebook news use was related to a modest over-time spiral of depolarization. Furthermore, we found that people who use Facebook for news were more likely to view both pro- and counter-attitudinal news in each wave”

Paper: Facebook news and (de)polarization: reinforcing spirals in the 2016 US election. Michael A. Beam, Myiah J. Hutchens & Jay D. Hmielowski. Information, Communication & Society,



11 chuck martel March 13, 2018 at 6:17 am

Americans don’t need to observe marches by those advocating genocidal ideologies. All they really need to do is read some history and spend part of their vacation driving around the internment camps known as “Indian reservations”.


12 Art Deco March 13, 2018 at 2:16 pm

the internment camps known as “Indian reservations”.

You never stop lying, do you?


13 rayward March 13, 2018 at 7:22 am

I suspect that the difference is between polarization and tribalism: when issues are presented as abstract concepts, there’s not that much polarization, but when issues are presented as two sides to the partisan (tribal) divide, there is. I would even make the case that on issues such as abortion the differences are more the result of tribalism than polarization. Thus, those opposed to abortion may state that it’s murder, but would overlook the fact that the leader of their tribe has paid for more than one. Did Germans in the 1930s become hyper-polarized (and mass murderers!) or did they simply follow their tribe? Good people are capable of horrible things. Indeed, the emphasis on polarization may well be a catalyst for horrible things. I will mention Singapore (as Cowen often does), which emphasizes social harmony even though Singapore is racially and religiously diverse. In the past, Americans practiced social harmony through shared experiences, in the neighborhoods they shared, in the churches they attended, in public schools where they were educated, in charitable causes they supported. Today, Americans are segregated, not only by race but by education, wealth, religion, and most importantly by geography. We are the tribe to which we belong.


14 chuck martel March 13, 2018 at 6:17 pm

The two largest “tribes’ in the US aren’t identified by their political ideologies even though they seem to agree with their contemporaries. The largest tribe is composed of public employees. The second largest tribe, and the most dedicated, is law enforcement. A third significant tribe is the career active duty military and its retirees. Self-identified Democrats and Republicans don’t constitute tribes.


15 Curt F. March 13, 2018 at 9:19 pm

So the largest three tribes are all public employees or former public employees? That’s a depressing thought.


16 Dick the Butcher March 13, 2018 at 7:51 am

When were we not polarized?

Let’s make a deal. You smart kids don’t tell me what I can do and what I can keep; and we’re cool.


17 Rehctub eht Kcid March 13, 2018 at 2:32 pm

A deal means you offer us something of equivalent value in return. You offer nothing.

Conclusively demonstrate that you utilize no public resources or infrastructure and that your actions and property have no impact or effect on anyone else, and we’ll talk.


18 Borjigid March 13, 2018 at 8:51 am

The MR Comments section continues to be exactly as polarized as we think!


19 Thomas March 13, 2018 at 12:45 pm

It’s tough not to be polarized when you skin color makes you a racist, your gender makes you a sexist, your location of birth makes you a xenophobe, redneck, ignoramus, and your opinion on whether we should remove statues makes you a Nazi. The full racist, sexist, xenophobic, scapegoating onslaught of the Democrat party is frightening.


20 Samoht March 13, 2018 at 2:21 pm

…and, once again, the pot calls the kettle black.

Quit whining, snowflake.


21 Art Deco March 13, 2018 at 2:31 pm

Quit whining, snowflake.

Then I’ll expect to hear no more about ‘white privilege’, ‘xenophobia’, ‘raaaacism’, &c. Thanks for your inspiration.


22 Borjigid March 13, 2018 at 2:34 pm

One person can’t be polarized- it takes two. For instance, me and you.


23 Thomas March 14, 2018 at 1:26 am

Yes I know that you are a racist, sexist, xenophobic Christianophobe. You are a mainstream progressive, consumed by hate.


24 Borjigid March 14, 2018 at 1:11 pm

And I would have gotten away with it, too, if it weren’t for you meddling kids!

25 Agra Brum March 13, 2018 at 5:23 pm

A perfect example of selection bias. We’re the ones who’ve chosen to argue online


26 Swimmy March 13, 2018 at 10:03 am

But the literature on increasing polarization doesn’t simply rely on opinion polls. DW-NOMINATE and similar methods have found increasing polarization in Congress. As Chris Posser points out above, response rates and polarization can correlate as well; people put off by increasing polarization drop out. They might drop out of elections as well as polls.


27 Anonymous March 13, 2018 at 11:04 am

Jerry Taylor tweeted a long nuts and bolts thread on current political beliefs a couple hours ago:

No idea how the 2000 people were found .. but I find the whole thing worrying. It’s just too detached from day to day problems.

It is almost as if (and I think we see that here) some don’t want to believe there are unusual day to day problems, because that would upset a deeply ingrained SOP. It is much easier for them to treat problems as insubstantial, partisan and same as always.


28 Anonymous March 13, 2018 at 11:49 am

Wow. Tillerson learned he was fired from the tweet.

Call that your newly updated, pretended, normal, and call out those *noticing* this daily decline of standards and expectations as the problem.


29 Anonymous March 13, 2018 at 12:40 pm

“Tillerson: Russia obviously poisoned the guy.

Trump: You’re fired.

Press: You fired him for saying that?

Trump: No, I fired him on Friday.

Tillerson aide: He wasn’t fired on Friday, it came as a complete surprise.

Trump: You’re fired too.”


30 Thomas March 13, 2018 at 12:46 pm

Apparently he was not fired but asked to resign on Friday. I know that dishonesty is your trade but c’mon.


31 Anonymous March 13, 2018 at 12:54 pm

You are behind on the news. As that clip shows, the State Department guy told the truth about that, and then he was fired.


32 Anonymous March 13, 2018 at 12:55 pm
33 Known Fact March 13, 2018 at 4:18 pm

If people weren’t so polarized they wouldn’t care so much about stuff like did Rex Tillerson know he was fired or not.

34 Anonymous March 13, 2018 at 4:33 pm


I feel like I should read you the riot act: You have forgotten what Constitutional government is supposed to be. You have forgotten the role of our Executive Branch. You have forgotten how an administration is supposed to work. You have forgotten what the Cabinet is supposed to be. You have forgotten how a Whitehouse is supposed to oversee all those things, with the President as chief executive.

You accept an angry old man, a random Fox viewer, who tweets what he feels, as a replacement for all those things.

It isn’t that Tillerson was fired, it is that yet again no one anywhere else in this “administration” knows what is going to happen until the tweet hits the fan.

35 Anonymous March 13, 2018 at 4:51 pm

The WaPo sees it similarly:

“Clearly, Trump does not want a secretary of state, or any Cabinet colleague, who ignores his tweets and gets on with the job. He wants someone who will respond to his constantly changing whims, who will cater to his moods. If he decides one week to insult the leader of North Korea, he doesn’t want to hear any objections. If he decides the next week that he wants to fly to Pyongyang, then he doesn’t want to hear any criticism of that either. He lives in a fantasy world of his own making, a nonstop television show in which he is the only star; he doesn’t want people who keep telling him that the real world looks somewhat different or that extravagant gestures might have severe consequences.”

36 Anonymous March 13, 2018 at 5:45 pm

Trump is sitting in my head, drinking my liquor, eating my snacks, watching my TV and I’m in the kitchen cooking his supper. It’s not at all fair.


37 Anonymous March 13, 2018 at 6:04 pm

That would be a sensible thing to say, if Trump were still doing television shows I didn’t watch.

But it’s kind of dumb as ..

Game Theory Scowls at Trump-North Korea Talks

Tyler’s concerns look that much greater with the new context.


38 Edward Burke March 13, 2018 at 11:21 am

Great: America itself is not nearly as polarized as our media elites might prefer us to think, instead, our elites themselves are deeply polarized and continue to try to steer public discourse in ways suitable to whatever ideological or economic or political motivation drives our divided, polarized elites.

I do enjoy the spectacle of cults of celebrity viciously attacking each other in public, but I wonder whether our academic elites themselves are plugged securely into whatever is going on these days.

How are complete apoliticization and wholesale political disengagement among both elites and “ordinary Americans” faring of late? Is political disengagement among all classes and cohorts becoming more widespread? Are Americans showing signs of deferring more, or less, to the “experts” of our befuddled elite classes?


39 DanC March 13, 2018 at 12:51 pm

I worry that the press has become polarized to the point they are dishonest. Not to mention that honest debate in politics has been replaced with the desire to find wedge issues that will drive fear or anger to electoral success.


40 Art Deco March 13, 2018 at 2:29 pm

I worry that the press has become polarized to the point they are dishonest.

Dan Rather’s career began in the 1950s.

honest debate in politics has been replaced

‘Honest’ ‘debate’? You mean like the fare offered on PBS NewsHour?

with the desire to find wedge issues that will drive fear or anger to electoral success.

Black Lives Matter is humbug. And it’s someone’s fault to boot.


41 DanC March 13, 2018 at 3:15 pm

Ok great analysis, excellent points.
Good use of negative space
No idea what it means but good work


42 Art Deco March 13, 2018 at 2:22 pm

instead, our elites themselves are deeply polarized

They’re hardly polarized at all, and differ primarily in the degree to which they take an interest in various social offenses. Our elites are divided between those pushing destructive schemes and those who cave into them. You have some resistance among elected officials, but not in any other professional establishment.


43 Agra Brum March 13, 2018 at 5:33 pm

It’s much more that most people want contradictory things. Strong social security, more government assistance with health care, a strong military, lots of social programs, a big infrastructure bill, and lower taxes. Those are all popular, and also impossible to provide every single one.

Abortions for some, miniature American flags for others.


44 Floccina March 13, 2018 at 4:48 pm

To me Democrats and republicans seem very similar it is me who is on the other pole.


45 8 March 16, 2018 at 1:23 pm

Even if 99% of Russians and Americans are not polarized, if the 1% who run the govt and military are more polarized, we get a war.


Leave a Comment

Previous post:

Next post: