Response rates are falling

In 1997, the response rate to a typical telephone poll was a healthy 36 percent, according to Pew. By 2012, it had fallen to 9 percent. Fortunately, many surveys appear to be doing a good job of weighting the answers of people who do respond, to make up for those who don’t. Still, the long-term reasons for concern are clear: People who are more likely to avoid polls, such as anyone born after, say, 1980, are different from those who answer them.

The response rate of the Labor Department’s monthly jobs survey is far higher (about 89 percent) than that of a political poll, but it has also fallen (from 96 percent in the 1980s). Not surprisingly, the people who do not respond have different experiences in the job market than those who do.

That is from David Leonhardt.  One implication is that actual unemployment may be higher than we are measuring.

Comments

This just means,

that after the official statistics, for e.g. unemployment became a joke after 1980,

the telephone based become so too, now

And your evidence for this is what exactly?

Comments for this post are closed

Comments for this post are closed

We put the phone down on all nuisance calls: if some should happen to be polling, hard cheese.

Yep, we do the same thing. I'd be willing to talk to pollsters if I weren't getting endless calls begging for money, trying to sell me stuff, looking to scam me, etc. But once you've fallen into the spam bucket, it only pays to pull you back out if I really want to talk to you.

Comments for this post are closed

Who still picks up the phone for an unidentified caller?

They are learning how to trick the caller ID.

I got one from myself the other day. I picked it up because I was curious to know who was calling with my number and my name.

Comments for this post are closed

It depends on if you are expecting a call from a number you may not know. For example, if you are job hunting you will likely accept a call from any number because it could be a call from some HR department or recruiter.

Comments for this post are closed

Comments for this post are closed

Comments for this post are closed

Why does it mean "that actual unemployment may be higher than we are measuring?" It might just as well mean the opposite.

That's what I thought. An unemployed person would more like be home to answer.

How does a homeless person manage to keep a landline with him all the time?

Very few people become homeless when they are unemployed. Even at the extreme most unemployed people end up living with relatives or friends who do have phones* (and the survey asks about household members, not just the individual contacted).
And on the other side of the coin, there are homeless people who do have jobs.

Also surveys are not barred from calling cell phones as telemarketers are.

* and of course many unemployed people have a working spouse who supports the household.

Correct, surveys are not barred from calling cell phones, but it is illegal to use autodialers to call a cellphone. Most surveys do not have people actually punching buttons on the phone, but use autodialers, so are in a legal grey area.

Comments for this post are closed

Chris, I would be surprised if government agencies are not exempt from those regulations.

Comments for this post are closed

Comments for this post are closed

Comments for this post are closed

Comments for this post are closed

Comments for this post are closed

The typical person who responds to an unsolicited call is probably different than a typical person who doesn't respond to those calls even after adjusting for demographics.

A 44 year old employed white mother living in Omaha who is too busy to handle a 5 minute phone poll probably has different political preferences than a 44 year old employed white mother living in Omaha who is NOT too busy to answer the phone.

No?

Certainly possible, but that's true of whatever instrument you use.

When push comes to shove, the only good poll is one whose model accurately outputs variables that are predictive of real world values, e.g. election results. Whether people who answer phones have different political preferences than those who do is almost irrelevant, except insofar as it helps you come up with the model in the first place.

My impression is that, if anything, good polls (as defined by those polls conducted by organizations that have accurately predicted elections in the past) have become more predictive of election results, not less. So, changing poll response rates doesn't seem to be an insurmountable challenge in building polling models.

Comments for this post are closed

Comments for this post are closed

Not to forget all those folks, who do not hav any more a landline number,

or grab their gun instead of answering their phone : - )

Comments for this post are closed

Economists estimate as opposed to measure unemployment. Measuring something implies a degree of precision compared to a known value. Instead of talking about what fluctuations in unemployment mean and what does/does not cause unemployment why don't economists put effort into developing a new method of estimating unemployment? With all the data available today is there not a more accurate method of measuring unemployment that doesn't involve calling 60k people on the phone like it's 1940 still?

+1

Comments for this post are closed

+1 There are probably many more predictive measures than surveys: google searches, electronic help wanted ads, etc.

Comments for this post are closed

Comments for this post are closed

The no call list has an exception for surveys. As a result, telemarketers have a habit of conducting "surveys" whose primary purpose is to sell you a product. As a result of this, people won't answer surveys any more.

The no call list didn't exist in 1997.

That's why.

The no-call list is a colossal failure.

More people than ever have signed up not to be called.

Fewer companies than ever have signed up to receive the numbers of people who do not want to be called.

Surveys or not, the whole enterprise should be abandoned.

Comments for this post are closed

Comments for this post are closed

Any survey which only covers people who have a landline will be heavily biased towards stable people. There are all kinds of groups who don't have a landline: itinerants, new immigrants, people on short-term or informal rented accommodation, the poor in general. If you're trying to measure social issues, missing out those groups will completely distort the numbers.

I just met a young chap who didn't have a mobile phone. The reason - he was freshly released from prison.

Comments for this post are closed

I do not know anyone under 50 with a landline. Do you?

Exactly. I was going to say that land lines aren't biased towards stable people, they're biased towards old people. Nobody I know has a land line aside from the office phone (which I'd get rid of too if they'd let me).

Comments for this post are closed

I do. All of the families of my daughter's friends do, to the best of my knowledge.

Comments for this post are closed

Comments for this post are closed

Why the assumption only land lines are called? Surveys can legally call a cell phone too. In fact, if you only have a cell phone databases will list the cell phone number as your phone number.

True, but back when I was a kid and we had a land line, when the phone rang someone picked it up. Today if my cell phone rings and it's not a number I recognize (particularly if it's an out-of-area code call) I let it go to voicemail. I generally don't talk to people on the phone anyway (aside from parents and boss -- everyone else texts) so I'm not likely to take a call from anyone I'm not already expecting to call. Granted this is personal anecdote, but in my experience many others have a similar approach so it wouldn't surprise me that getting people to answer their phones is much harder now than in the past.

As I also commented above, people who are expecting a call from a number they do not know (or have stored in their phone) will probably answer any call that comes in. So there are people who answer phone calls whose numbers they do not recognize.

Comments for this post are closed

Comments for this post are closed

Comments for this post are closed

Comments for this post are closed

Who really believes the response rate claimed by the Department of Labor? Even the 89% figure claimed for today seems improbable to me.

I see no reason to doubt it. And no "paranoia" is not an reason.

Comments for this post are closed

There's been a case of a whistleblower reporting being pressured to fake results. Of course there's every reason to doubt anybody's claim to get an 89% response rate.

I suspect though the figure is misreported. I'm guessing what is meant is, 89% of those who agree to respond to a series of surveys actually respond to each individual survey. The proportion of those who agree to respond to a series of surveys from among those who are asked to is presumably more like the private surveyors' 9%.

Comments for this post are closed

Comments for this post are closed

Comments for this post are closed

Blame push polls. Your attitude only needs to turn from "yes, I'd love to help you advance the social sciences" to "bug off, you manipulative telemarketer" a few times before you realize it's a big time saver to skip the first step.

Comments for this post are closed

I am highly dubious of this result. First, the people actually creating the estimates all have PhDs with specialties in the correction for nonresponse bias. And they're in the gov't so they have a lot of levers at theirt disposal to do those corrections (tax returns, employer stats, etc.) Sure, it's still an art, but the more data you have at your disposal from indepedent (non-survey-based) sources, the better your chances. In addition, the conclusion here (that unemployment rates are understated) is belied by the analysis you just linked to in the FT-Alphaville analysis in the previous posting which pointed out that the survey data and the non-survey data lined up.

It's not a result; it's just speculation about the effect on the unemployment rate.

But beyond the PhD qualifications, we have the fact that it's not the unemployment rate (level) that's at issue, it's the change in the rate (trend) that's more important. This means there's a huge bias towards not changing the methodology, so that the current results are comparable to previous results.

This is mostly a good tendency, but not helpful as the old categories slowly change. (e.g. lots of people "employed" but unable to get 40 hours, discouraged workers not counting, increase in disability classifications, etc.)

Comments for this post are closed

Comments for this post are closed

Like Dearieme, my wife and I don't answer telephone calls anymore. If it's important, they'll call back. :-)

Comments for this post are closed

Zuckerberg knows all the answers anyone born after 1980 would give anyway.

Off on a tangent here but this remark reminded me of something I read in a New Yorker article about the OK Cupid founder...
Apparently algorithms can already tell, just by using your likes on Facebook (not using status updates or messages), with 88% accuracy if you are straight or gay. With 66% accuracy they can tell if your parents divorced before you turned 21 using the same data.

I'm a mobile marketer and major spender on Facebook ads. You would blanch in terror if you knew what we know about you. O_O

I believe it, which is why I don't have a Facebook account. Me FTW!

Of course my wife does so you probably have me figured out anyway.

Comments for this post are closed

Comments for this post are closed

Comments for this post are closed

Comments for this post are closed

What did everybody think of the fact that the people get called for four months, then not for eight months, then for four months again, for a total of 16 months' participation?

Here's a more detailed explanation of what is going on with this polling:

"Over time, the kinds of answers that people give — or the kinds of people who respond — change. In later months as part of the survey panel, people who aren’t working are less likely to report being available to work and having looked for a job in the previous four weeks, which is the definition for unemployment. The differences are big, too.

The unemployment rate in the first half of 2014 among people in the first month of being interviewed was 7.5 percent. Among people in the final month of being interviewed, it was only 6.1 percent. Because the Labor Department weights later panelists – for whom there is historical data – more heavily, the official unemployment rate during this period was 6.5 percent.

That number seems too low. The authors note that the higher jobless rate among early-month panelists correlates more strongly with some other economic indicators than the rate among later-month panelists."

So how close is the 6.5% official unemployment figure for the first half of 2014 to the truth?

I suspect it is in the ballpark, if you accept the definition of unemployed as someone not working but actively looking for work. And the direction we see in the unemployment rate (going up or down) is likely to be accurate even if the number itself has some uncertainty about it (as it certainly does), since the factors creating the uncertainty will be essentially random and not change from one survey to the next.

Comments for this post are closed

Comments for this post are closed

This is why it is good news when governments cut wasteful spending in their statistical services.

Learn from Canada. We've been really good at that for about 8 years.

Building the future, one cut at a time. Drill, baby drill. Non-renewables will be the economy of the future.

Comments for this post are closed

Hi Tyler,

Is it possible to get in touch with you directly by email? I'm working on the promotion of the latest film by Harold Crooks on tax havens and by seeing your posts I think that you'll be interested.
Please see here: https://www.facebook.com/pricewepay.thefilm.

Regards,
Melkiades

Comments for this post are closed

Comments for this post are closed