Department of spurious correlations?

by on March 7, 2013 at 2:15 am in Economics, Science | Permalink

Here is the abstract of a forthcoming AER piece, written by M. Keith Chen:

Languages differ widely in the ways they encode time. I test the hypothesis that languages that grammatically associate the future and the present, foster future-oriented behavior. This prediction arises naturally when well-documented effects of language structure are merged with models of intertemporal choice. Empirically, I find that speakers of such languages: save more, retire with more wealth, smoke less, practice safer sex, and are less obese. This holds both across countries and within countries when comparing demographically similar native households. The evidence does not support the most obvious forms of common causation. I discuss implications for theories of intertemporal choice.

Here is from a recent article in The Chronicle of Higher Education, by Geoffrey Pullum:

Chen’s data on languages comes from the World Atlas of Language Structures (WALS), and his evidence on prudence from the World Values Survey (WVS). Both are fully Web-accessible. Sean Roberts, who studies language evolution at the Max Planck Institute for Psycholinguistics in Nijmegen, decided to investigate the other linguistic factors treated in WALS to see how they related to prudence. He compared the goodness of fit for linear regressions on each of a long list of properties of languages (the independent variables), using as the dependent variable the answers that speakers gave to the WVS question “Did you save money last year?”

The results (see this blog post for an informal account) were jaw-dropping. He found that dozens of linguistic variables were better predictors of prudence than future marking: whether the language has uvular consonants; verbal agreement of particular types; relative clauses following nouns; double-accusative constructions; preposed interrogative phrases; and so on—a motley collection of factors that no one could plausibly connect to 401(k) contributions or junk-food consumption.

There is a bit more here.

For the pointer I thank Mike T.  And I would gladly run a response from Chen, if he has interest in drafting one.

Addendum: Here is an important update from the critic, after improving the specification of his alternative fits:

The results showed that there was only one other linguistic variable that improved the fit of the model more than future tense.  That is, future tense was a better predictor than 99% of the linguistic variables.  For comparison, Dediu & Ladd’s test of the link between linguistic tone and Microcephalin/ASPM found that the hypothesised link was stronger than 98.5% of many thousands of links between genetic and linguistic factors.

Brent March 7, 2013 at 2:27 am

Okay, but one has an inkling of causal theory behind its empirical correlation… while the others seem a bit lacking in the former.

LB March 7, 2013 at 2:36 am

That’s true, but most alleged Whorfian effects in other domains of cognition are tiny or have turned out to be non-reproducible. For the latter category, see Al Bloom’s work from the 1980′s on counterfactual thinking in Chinese. I won’t explicitly comment on more recent work coming out of some high-profile labs — you can look up the non-replications easily enough. I don’t think that our prior should be very high on this hypothesis.

Brent March 7, 2013 at 3:10 am

No disagreement about the conclusion to be highly suspicious… But for the reasons you mentioned and not because other random language variables happen to be correlated.

Dan March 7, 2013 at 3:24 am

So you have to rely on intuitive sentiments to know if the empirical data are worth anything?

Brent March 7, 2013 at 5:51 am

Is there an alternative to having a causal theory? Would love to hear it.

Rasa Karapandza March 7, 2013 at 3:17 am

The author that disputed Chan’s paper has redone his analysis. He recognised that he was wrong and that Chan’s findings were much more robust than he previously claimed.
http://www.replicatedtypo.com/whorfian-economics-reconsidered-residuals-and-causal-graphs/6011.html

What I find disturbing in the AER publication though is that it fails to cite some previous work on a similar topic:
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1910011

Chris March 7, 2013 at 8:29 am

This seems to take quite a bit of wind out of the critique. More of an oh-nevermind than a simple update. I hate announcing findings and then doing an oh-nevermind, happened too much in my early career I’m a triple-checker these days.

mavery March 7, 2013 at 11:55 am

Yeah, this sort of undermines the entire thesis of the critique. I mean, the addendum actually seems to me to be evidence in favor of the original claim rather than evidence against it.

Axa March 7, 2013 at 6:41 am

my goodness, mexicans would be rich. they always talk about resting today and doing the work “mañana”. there’s a really strong dissociation between present and future for sure. where are the savings?

IVV March 7, 2013 at 8:35 am

You haven’t worked with Mexicans up close and personal, have you?

Pseudonymous March 7, 2013 at 10:23 am

You’re mixing up the direction of causality, in Chen’s model dissociation causes less savings, because the future period is ‘farther’ away, so you discount it more heavily.

Bill March 7, 2013 at 7:08 am

I have actually done a similar textual analysis using the blog postings from this site.

I tested the hypothesis that predictive economic comments were inversely correlated with actual outcomes, that postings with titles ending in question marks universally give an out to the poster later when the comments point the other way, and that Tyler uses more future tense verbs than Alex, but that Alex uses more nouns involving body parts, drugs, and regulation.

Finally, mentioning Krugman increased the number of comments and comment length by 50%.

Andrew' March 7, 2013 at 7:57 am

That’s some fine work Bill. I never would have guessed that a real-time debate would increase comments over flying puffins.

John B. Chilton March 7, 2013 at 9:52 am
John B. Chilton March 7, 2013 at 10:30 am

That should be Sean Roberts, not Pullum.

Miley Cyrax March 7, 2013 at 9:57 am

This is pretty funny, that the brouhaha is over whether this supports the Sapir Whorf hypothesis or not. It’s like these people forget that language is indisputably the manisfestation of thought, although there may be a certain degree of feedback the opposite way. But then that might lead them to entertain icky notions like the possibility that high time preference populations develop and/or maintain languages with more future-centric words.

Knee jerk cries of “correlation is not causation!” is a cliche of its own.

Miley Cyrax March 7, 2013 at 10:02 am

Should be “But then remembering that might lead them to entertain…” and low time preference, not high. This is what I get for having high time preference and hitting “submit” too quickly.

Pseudonymous March 7, 2013 at 10:56 am

I’m not sure I buy Pullum’s ‘diceyness’ about the causal intuition. The possibility of two causal effects with opposing directions isn’t new to economics. Doesn’t the same apply when we think about the substitution and income effects? I don’t entirely buy Chen’s causal explanation, either, but I don’t think Pullum’s idea that there’s a lot of alternatives is necessarily a great critique.

Troy Camplin March 7, 2013 at 4:57 pm

Two things. 1) Sapir-Whorf correlations have never panned out, and translation would be impossible if the theory were true, and 2) nevertheless, I must point out that just because one can find a number of other (spurious) correlations that are stronger, that does not mean that the thing being studied does not have a causal linkage, however slight.

Back to 1, thinking is not language, and language is not thought, even if some thinking can be done in language.

Carola Binder March 8, 2013 at 12:17 am

Another article relating language to economics, and also using the WALS data, is “Does language shape our economy? Female/male grammatical distinctions and gender economics.” They find “new empirical evidence that gender distinctions in language are strongly correlated with female labour-force participation and the use of gender political quotas.” The link is:
http://www.voxeu.org/article/language-matters-gender-grammar-and-observed-gender-discrimination

Comments on this entry are closed.

Previous post:

Next post: