Beware of persistence studies

A large literature on persistence finds that many modern outcomes strongly reflect characteristics of the same places in the distant past. However, alongside unusually high t statistics, these regressions display severe spatial auto-correlation in residuals, and the purpose of this paper is to examine whether these two properties might be connected. We start by running artificial regressions where both variables are spatial noise and find that, even for modest ranges of spatial correlation between points, t statistics become severely inflated leading to significance levels that are in error by several orders of magnitude. We analyse 27 persistence studies in leading journals and find that in most cases if we replace the main explanatory variable with spatial noise the fit of the regression commonly improves; and if we replace the dependent variable with spatial noise, the persistence variable can still explain it at high significance levels. We can predict in advance which persistence results might be the outcome of fitting spatial noise from the degree of spatial au-tocorrelation in their residuals measured by a standard Moran statistic. Our findings suggest that the results of persistence studies, and of spatial regressions more generally, might be treated with some caution in the absence of reported Moran statistics and noise simulations.

That is from a new paper by Morgan Kelly, and here is a look at the studies he considers, via Morton Jerven.

Comments

That's a lot of $10 words to say Hillary is a nasty woman.

LOL. I thought something similar. I thought it was an entire paragraph of gobbledy gook that essentially said statistical conclusions were useless and wrong.

But who on earth would expect useful information to emerge from regression analyses of poor quality data?

Though it's no doubt a useful contribution to demonstrate that a sensible scepticism would have been well advised.

Yeah, but what if poor quality data is all you have?

If the quality is poor enough go and study something else.

Persistence studies. One learns something everyday reading this blog. I googled the term and up popped all kinds of papers relating to persistence studies, most related to clinical persistence studies. Here's one that informs the "left-censored bias" (I thought readers of this blog would appreciate that) of persistence studies: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3901829/

Persistence studies are often cited, including here, because results seem novel.

Perhaps too novel.

An abstract written to the specifications of Georg Hegel's Guide To Writing And Composition.

I don't ask for the moon, I'd just be obliged if anyone cares to mansplain to me the replies in the linked twitter thread - e.g. "Returning to the sign issue, the Standard Errors of Persistence simulations are building maybe half of the 'significant' fake results in worlds that get the signs wrong... and still counting that as a win."

Any econ translators out there care to explain whether the conjecture here is:

a) deep persistence (in the cases identified) doesn't actually exist and is statistically dubious. "Deep roots are not deep. Your lying eyes again!".
b) deep persistence does exist but it's not most parsimonious explained by "cultural persistence" itself but some correlated thing?
c) none of the above

b) is perhaps possible (do all those correlations with the Western Catholic Church really add up?), but it's hard to see a) being true, at least in the case of "Deep roots"...

This paper doesn't attempt to directly show that the deep persistence results go away when you redo various results 'right', but that you can get similar results by doing it wrong on faked data which looks like specific real data. So it's indirect: it casts doubt but it can't directly say that deep persistence doesn't exist (maybe all that spatial autocorrelation is *because* of deep persistence!) or that deep persistence is not cultural but something else (like genetic). If you were skeptical of deep persistence, this reduces the evidential value of the literature, but it doesn't provide evidence for the opposite.

If you had to pick one option, I would go with (A), I think: deep persistence isn't an independent thing above and beyond the trivial observation that countries which are rich in the past tend to be rich in the future, and rich countries tend to be near other rich countries. So, an example of "Galton's problem".

Hmmm... Then that moves the argument to 'deep persistence' advocates to demonstrate that the data is not wrong (and would be impossible to fake). It doesn't really matter if the same patterns can emerge from spurious, wrong data, if the data is correct; it just means you can't take that the models themselves make sense as a sort of confirmation of the data.

Though is it truly trivial to observe that countries and regions (and their 'offshoots', surely) which are rich in the 'deep' past tend to be rich in the future? It seems that many people would hold that this would not be so (world convergence is inevitable?). If it were completely uncontroversial that those that were rich or economically complex in 1600 will still likely be in 2100, and this was a baseline assumption that would seem trivial but it does not seem so.

this seems consistent with the highly dubious use of the "path dependence" concept in almost every context it is applied

If an effect is too uncomfortable for my town, my town will not be there for historians to notice.

If historians see a cluster, then they are see a cluster of people willing to tolerate bad stuff to keep their town intact. So there is a real problem to separating tolerance for bad with the level of dislike. The town either stays or goes, it is sort of a binary decision that loses a whole bunch of data for the historian.

Interesting paper with really nice visuals.

Wish they would have focused on the bigger problem in this literature tho... nearly all of the parameter estimates in the lit suffer from bias. It’s great they want accurate t-stats but that’s not much comfort when almost all the estimates are biased.

In cases where there is omitted variation in what’s being measured (imperfect proxy variables) and the omitted portion exhibits correlation with the included portion, then least squares type estimators will be biased (basically every study is presenting biased parameter estimates).

One would need to include spatial lags of both the dependent and independent variables in this case (spatial durbin models).

In this case, the responses (dy/dx) take the form of n x n matrices that facilitate spatial spillover impacts from a change in an x in place i on y in place j. The main diagonals would be the direct effects and the off diagonals the spatial spillovers.

So the real problem in this literature as I see it is bias due to mispecification caused by imperfect proxies leading to omitted variation that is correlated with the included variation.

I don’t know why this problem is so little noticed. Starting to think I am just mistaken in my thinking...

In an effort to make the point a little better. Suppose we are trying to estimate y=xB + e using spatial data. Suppose further that x doesn’t really capture all the variation about something that the researchers are using it to. In this case, we should be estimating y=xB + x’B’. However, x’ is not measureable, otherwise it could just be included. If x’ is uncorrelated with x and is spherical, then x’B functions as a disturbance term e, and we are good to go as is.

Now its likely that x is correlated with x’ but is unobserved (the observed and unobserved portions of some thing are likely to be correlated with each other). Given we are using spatial data, suppose x’ follows a spatial autoregressive process like x’=pWx’ + d. Rewrite it as d(In – pW)^-1. Here, p is a scalar parameter reflecting the strength of the spatial correlation, d is a vector of disturbances that follow a normal with mean of 0 and constant variance. W is an n x n spatial weight matrix with non-zero values in positions reflecting neighboring points or places. Substituting the rewrite into y=xB+x’B’, you get y=xB + B’d(In – pW)^-1. Here we can see that B’ has the effect of increasing the variance of d. To make this less messy, I will write B’d as u. y=xB+u(In – pW)^-1. <- [lets denote this as (1)].

If x’ is correlated with x, then x and u are correlated. Writing it as a linear correlation, it could be written u=xZ + q (where q is again assumed normal with mean 0 and constant variance).

Now substitute this into (1) and we get a data generating process of y=xB+(In – pW)^-1(xZ + q) which we can rewrite as y=XB + (In – pW)^-1xZ + (In – pW)^-1q. Here, B is not unbiased due to the linear dependence of u on x. Multiply through by (In – pW) and resolve for y and you get y=pWy + x(B + Z) + Wx(-pB) + q. This is a spatial durbin model that includes spatial lags of both y and x.

So if x is an imperfect proxy and the omitted portion of it (x’) is correlated with the included portion, then a spatial dubin model should be estimated to get unbiased estimates of B.

Furthermore, response of y to changes in x (dy/dx) will take the form of n x n matrices with the main diagonals being the direct effects and the off diagonals the indirect spatial spillover effects.

So the problem isn’t just about t-stats, its about bias.

"I don’t know why this problem is so little noticed."

I agree, although I must quickly add that spatial econometrics is not my field, so maybe this has been addressed before in the literature (but Kelly's article says no).

At one point in the paper Kelly draws the obvious analogy with the absurdly high R2 statistics and spurious correlations one can get with time series data. That problem is well known; why hasn't this spatial autocorrelation problem been addressed before?

The paper recommends calculating a Moran statistic, analogous to calculating a Durbin-Watson statistic after doing OLS to check for autocorrelation. Seems like solid advice.

If I'm understanding what you're suggesting above, it looks analogous to a standard solution for time serial autocorrelation: take first differences of the data, with respect to time when we have serial autocorrelation -- or with respect to distance when we have spatial autocorrelation?

With regard to the questions that other commenters have asked: yes, Kelly's research calls into question previous results that looked at persistence. But for a specific and seemingly fundamental and widely applicable reason, that Student addresses in his comments. And I think the more interesting and important discussion of Kelly's results is contained in those comments.

This could change the standard ways that people analyze spatial data, by routinely doing Moran tests if nothing else ... if I'm understanding the article, again it's not a field that I've studied.

It is good advice to check for spatial autocorrelation. Just saying that many people are not understanding the nature of the problem and as a result, are presenting biased estimates, not just bad t-stats. Likewise, they misunderstanding how to interpret the partials by failing to account for indirect spatial spillovers.

Comments for this post are closed