Response rates are falling

by on August 27, 2014 at 1:57 pm in Data Source, Economics, Uncategorized | Permalink

In 1997, the response rate to a typical telephone poll was a healthy 36 percent, according to Pew. By 2012, it had fallen to 9 percent. Fortunately, many surveys appear to be doing a good job of weighting the answers of people who do respond, to make up for those who don’t. Still, the long-term reasons for concern are clear: People who are more likely to avoid polls, such as anyone born after, say, 1980, are different from those who answer them.

The response rate of the Labor Department’s monthly jobs survey is far higher (about 89 percent) than that of a political poll, but it has also fallen (from 96 percent in the 1980s). Not surprisingly, the people who do not respond have different experiences in the job market than those who do.

That is from David Leonhardt.  One implication is that actual unemployment may be higher than we are measuring.

genauer August 27, 2014 at 2:05 pm

This just means,

that after the official statistics, for e.g. unemployment became a joke after 1980,

the telephone based become so too, now

JonFraz August 27, 2014 at 6:54 pm

And your evidence for this is what exactly?

dearieme August 27, 2014 at 2:09 pm

We put the phone down on all nuisance calls: if some should happen to be polling, hard cheese.

albatross August 27, 2014 at 10:14 pm

Yep, we do the same thing. I’d be willing to talk to pollsters if I weren’t getting endless calls begging for money, trying to sell me stuff, looking to scam me, etc. But once you’ve fallen into the spam bucket, it only pays to pull you back out if I really want to talk to you.

Mitch Berkson August 28, 2014 at 8:34 am

Who still picks up the phone for an unidentified caller?

randomworker August 28, 2014 at 8:44 am

They are learning how to trick the caller ID.

I got one from myself the other day. I picked it up because I was curious to know who was calling with my number and my name.

JOnFraz August 28, 2014 at 2:12 pm

It depends on if you are expecting a call from a number you may not know. For example, if you are job hunting you will likely accept a call from any number because it could be a call from some HR department or recruiter.

Marc Passy August 27, 2014 at 2:24 pm

Why does it mean “that actual unemployment may be higher than we are measuring?” It might just as well mean the opposite.

TMC August 27, 2014 at 5:43 pm

That’s what I thought. An unemployed person would more like be home to answer.

mulp August 27, 2014 at 6:26 pm

How does a homeless person manage to keep a landline with him all the time?

JonFraz August 27, 2014 at 6:57 pm

Very few people become homeless when they are unemployed. Even at the extreme most unemployed people end up living with relatives or friends who do have phones* (and the survey asks about household members, not just the individual contacted).
And on the other side of the coin, there are homeless people who do have jobs.

Also surveys are not barred from calling cell phones as telemarketers are.

* and of course many unemployed people have a working spouse who supports the household.

Chris S August 27, 2014 at 10:41 pm

Correct, surveys are not barred from calling cell phones, but it is illegal to use autodialers to call a cellphone. Most surveys do not have people actually punching buttons on the phone, but use autodialers, so are in a legal grey area.

JOnFraz August 28, 2014 at 2:13 pm

Chris, I would be surprised if government agencies are not exempt from those regulations.

Cooper August 27, 2014 at 3:08 pm

The typical person who responds to an unsolicited call is probably different than a typical person who doesn’t respond to those calls even after adjusting for demographics.

A 44 year old employed white mother living in Omaha who is too busy to handle a 5 minute phone poll probably has different political preferences than a 44 year old employed white mother living in Omaha who is NOT too busy to answer the phone.


Zephyrus August 27, 2014 at 3:49 pm

Certainly possible, but that’s true of whatever instrument you use.

When push comes to shove, the only good poll is one whose model accurately outputs variables that are predictive of real world values, e.g. election results. Whether people who answer phones have different political preferences than those who do is almost irrelevant, except insofar as it helps you come up with the model in the first place.

My impression is that, if anything, good polls (as defined by those polls conducted by organizations that have accurately predicted elections in the past) have become more predictive of election results, not less. So, changing poll response rates doesn’t seem to be an insurmountable challenge in building polling models.

genauer August 27, 2014 at 3:20 pm

Not to forget all those folks, who do not hav any more a landline number,

or grab their gun instead of answering their phone : – )

Matt V August 27, 2014 at 3:57 pm

Economists estimate as opposed to measure unemployment. Measuring something implies a degree of precision compared to a known value. Instead of talking about what fluctuations in unemployment mean and what does/does not cause unemployment why don’t economists put effort into developing a new method of estimating unemployment? With all the data available today is there not a more accurate method of measuring unemployment that doesn’t involve calling 60k people on the phone like it’s 1940 still?

Timber Tom August 27, 2014 at 4:20 pm


Bill August 28, 2014 at 10:30 am

+1 There are probably many more predictive measures than surveys: google searches, electronic help wanted ads, etc.

Ken Arromdee August 27, 2014 at 4:04 pm

The no call list has an exception for surveys. As a result, telemarketers have a habit of conducting “surveys” whose primary purpose is to sell you a product. As a result of this, people won’t answer surveys any more.

The no call list didn’t exist in 1997.

That’s why.

JosieB August 28, 2014 at 1:13 am

The no-call list is a colossal failure.

More people than ever have signed up not to be called.

Fewer companies than ever have signed up to receive the numbers of people who do not want to be called.

Surveys or not, the whole enterprise should be abandoned.

Andrew M August 27, 2014 at 4:15 pm

Any survey which only covers people who have a landline will be heavily biased towards stable people. There are all kinds of groups who don’t have a landline: itinerants, new immigrants, people on short-term or informal rented accommodation, the poor in general. If you’re trying to measure social issues, missing out those groups will completely distort the numbers.

Timber Tom August 27, 2014 at 4:21 pm

I just met a young chap who didn’t have a mobile phone. The reason – he was freshly released from prison.

Benny Lava August 27, 2014 at 5:12 pm

I do not know anyone under 50 with a landline. Do you?

Brian August 28, 2014 at 11:28 am

Exactly. I was going to say that land lines aren’t biased towards stable people, they’re biased towards old people. Nobody I know has a land line aside from the office phone (which I’d get rid of too if they’d let me).

Careless September 1, 2014 at 12:10 am

I do. All of the families of my daughter’s friends do, to the best of my knowledge.

JonFraz August 27, 2014 at 6:59 pm

Why the assumption only land lines are called? Surveys can legally call a cell phone too. In fact, if you only have a cell phone databases will list the cell phone number as your phone number.

Brian August 28, 2014 at 11:35 am

True, but back when I was a kid and we had a land line, when the phone rang someone picked it up. Today if my cell phone rings and it’s not a number I recognize (particularly if it’s an out-of-area code call) I let it go to voicemail. I generally don’t talk to people on the phone anyway (aside from parents and boss — everyone else texts) so I’m not likely to take a call from anyone I’m not already expecting to call. Granted this is personal anecdote, but in my experience many others have a similar approach so it wouldn’t surprise me that getting people to answer their phones is much harder now than in the past.

JOnFraz August 28, 2014 at 2:16 pm

As I also commented above, people who are expecting a call from a number they do not know (or have stored in their phone) will probably answer any call that comes in. So there are people who answer phone calls whose numbers they do not recognize.

Yancey Ward August 27, 2014 at 4:21 pm

Who really believes the response rate claimed by the Department of Labor? Even the 89% figure claimed for today seems improbable to me.

JonFraz August 27, 2014 at 7:00 pm

I see no reason to doubt it. And no “paranoia” is not an reason.

Tom August 28, 2014 at 8:40 am

There’s been a case of a whistleblower reporting being pressured to fake results. Of course there’s every reason to doubt anybody’s claim to get an 89% response rate.

Tom August 28, 2014 at 8:45 am

I suspect though the figure is misreported. I’m guessing what is meant is, 89% of those who agree to respond to a series of surveys actually respond to each individual survey. The proportion of those who agree to respond to a series of surveys from among those who are asked to is presumably more like the private surveyors’ 9%.

roystgnr August 27, 2014 at 4:29 pm

Blame push polls. Your attitude only needs to turn from “yes, I’d love to help you advance the social sciences” to “bug off, you manipulative telemarketer” a few times before you realize it’s a big time saver to skip the first step.

Jonathan August 27, 2014 at 4:39 pm

I am highly dubious of this result. First, the people actually creating the estimates all have PhDs with specialties in the correction for nonresponse bias. And they’re in the gov’t so they have a lot of levers at theirt disposal to do those corrections (tax returns, employer stats, etc.) Sure, it’s still an art, but the more data you have at your disposal from indepedent (non-survey-based) sources, the better your chances. In addition, the conclusion here (that unemployment rates are understated) is belied by the analysis you just linked to in the FT-Alphaville analysis in the previous posting which pointed out that the survey data and the non-survey data lined up.

zbicyclist August 27, 2014 at 11:49 pm

It’s not a result; it’s just speculation about the effect on the unemployment rate.

But beyond the PhD qualifications, we have the fact that it’s not the unemployment rate (level) that’s at issue, it’s the change in the rate (trend) that’s more important. This means there’s a huge bias towards not changing the methodology, so that the current results are comparable to previous results.

This is mostly a good tendency, but not helpful as the old categories slowly change. (e.g. lots of people “employed” but unable to get 40 hours, discouraged workers not counting, increase in disability classifications, etc.)

Dismalist August 27, 2014 at 4:48 pm

Like Dearieme, my wife and I don’t answer telephone calls anymore. If it’s important, they’ll call back. :-)

Pshrnk August 27, 2014 at 5:35 pm

Zuckerberg knows all the answers anyone born after 1980 would give anyway.

msgkings August 27, 2014 at 8:06 pm

Off on a tangent here but this remark reminded me of something I read in a New Yorker article about the OK Cupid founder…
Apparently algorithms can already tell, just by using your likes on Facebook (not using status updates or messages), with 88% accuracy if you are straight or gay. With 66% accuracy they can tell if your parents divorced before you turned 21 using the same data.

TJC August 27, 2014 at 8:18 pm

I’m a mobile marketer and major spender on Facebook ads. You would blanch in terror if you knew what we know about you. O_O

msgkings August 27, 2014 at 9:02 pm

I believe it, which is why I don’t have a Facebook account. Me FTW!

Of course my wife does so you probably have me figured out anyway.

jerseycityjoan August 28, 2014 at 1:41 am

What did everybody think of the fact that the people get called for four months, then not for eight months, then for four months again, for a total of 16 months’ participation?

Here’s a more detailed explanation of what is going on with this polling:

“Over time, the kinds of answers that people give — or the kinds of people who respond — change. In later months as part of the survey panel, people who aren’t working are less likely to report being available to work and having looked for a job in the previous four weeks, which is the definition for unemployment. The differences are big, too.

The unemployment rate in the first half of 2014 among people in the first month of being interviewed was 7.5 percent. Among people in the final month of being interviewed, it was only 6.1 percent. Because the Labor Department weights later panelists – for whom there is historical data – more heavily, the official unemployment rate during this period was 6.5 percent.

That number seems too low. The authors note that the higher jobless rate among early-month panelists correlates more strongly with some other economic indicators than the rate among later-month panelists.”

So how close is the 6.5% official unemployment figure for the first half of 2014 to the truth?

JOnFraz August 28, 2014 at 2:20 pm

I suspect it is in the ballpark, if you accept the definition of unemployed as someone not working but actively looking for work. And the direction we see in the unemployment rate (going up or down) is likely to be accurate even if the number itself has some uncertainty about it (as it certainly does), since the factors creating the uncertainty will be essentially random and not change from one survey to the next.

Nathan W August 28, 2014 at 3:11 pm

This is why it is good news when governments cut wasteful spending in their statistical services.

Learn from Canada. We’ve been really good at that for about 8 years.

Building the future, one cut at a time. Drill, baby drill. Non-renewables will be the economy of the future.

Melkiades August 30, 2014 at 9:50 am

Hi Tyler,

Is it possible to get in touch with you directly by email? I’m working on the promotion of the latest film by Harold Crooks on tax havens and by seeing your posts I think that you’ll be interested.
Please see here:


Comments on this entry are closed.

Previous post:

Next post: