Why the New Pollution Literature is Credible
My recent post, Air Pollution Reduces Health and Wealth drew some pushback in the comments, some justified, some not, on whether the results of these studies are not subject to p-hacking, forking gardens and the replication crisis. Sure, of course, some of them are. Andrew Gelman, for example, has some justified doubt about the air filters and classroom study. Nevertheless, I don’t think that skepticism about the general thrust of the results is justified. Why not?
First, go back to my post Why Most Published Research Findings are False and note the list of credibility checks. For example, my rule is trust literatures not papers and the new pollution literature is showing consistent and significant negative effects of pollution on health and wealth. Some might respond that the entire literature is biased for reasons of political correctness or some such and sure, maybe. But then what evidence would be convincing? Is skepticism then justified or merely mood affiliation? And when it comes to action should we regard someone’s prior convictions (how were those formed?) as more accurate then a large, well-published scientific literature?
It’s not just that the literature is large, however, it’s that the literature is consistent in a way that many studies in say social psychology were not. In social psychology, for example, there were many tests of entirely different hypotheses–power posing, priming, stereotype threat–and most of these failed to replicate. But in the pollution literature we have many tests of the same hypotheses. We have, for example, studies showing that pollution reduces the quality of chess moves in high-stakes matches, that it reduces worker productivity in Chinese call-centers, and that it reduces test scores in American and in British schools. Note that these studies are from different researchers studying different times and places using different methods but they are all testing the same hypothesis, namely that pollution reduces cognitive ability. Thus, each of these studies is a kind of replication–like showing price controls led to shortages in many different times and places.
Another feature in favor of the air pollution literature is that the hypothesis that pollution can have negative effects on health and cognition wasn’t invented yesterday along with the test (we came up with a new theory and tested it and guess what, it works!). The Romans, for example, noted the negative effect of air pollution on health. There’s a reason why people with lung disease move to the countryside and always have.
I also noted in Why Most Published Research Findings are False that multiple sources and types of evidence are desirable. The pollution literature satisfies this desideratum. Aside from multiple empirical studies, the pollution hypothesis is also consistent with plausible mechanisms and it is consistent with the empirical and experimental literature on pollution and plants and pollution and animals. See also OpenPhilanthropy’s careful summary.
Moreover, there is a clear dose-response effect–so much so that when it comes to “extreme” pollution few people doubt the hypothesis. Does anyone doubt, for example, that an infant born in Delhi, India–one of the most polluted cities in the world–is more likely to die young than if the same infant grew up (all else equal) in Wellington, New Zealand–one of the least polluted cities in the world? People accept that “extreme” pollution creates debilitating effects but they take extreme to mean ‘more than what I am used to’. That’s not scientific. In the future, people will think that the levels of pollution we experience today are extreme, just as we wonder how people could put up with London Fog.
What is new about the new pollution literature is more credible methods and bigger data and what the literature shows is that the effects of pollution are larger than we thought at lower levels than we thought. But we should expect to find smaller effects with better methods and bigger data. (Note that this isn’t guaranteed, there could be positive effects of pollution at lower levels, but it isn’t surprising that what we are seeing so far is negative effects at levels previously considered acceptable.)
Thus, while I have no doubt that some of the papers in the new pollution literature are in error, I also think that the large number of high quality papers from different times and places which are broadly consistent with one another and also consistent with what we know about human physiology and particulate matter and also consistent with the literature on the effects of pollution on animals and plants and also consistent with a dose-response relationship suggest that we take this literature and its conclusion that air pollution has significant negative effects on health and wealth very seriously.