Which published results can you trust?

That is the theme of my latest Free Press column, starting with the recent Oliver Sacks debacle.  Here is one excerpt:

…as my George Mason University colleague Bryan Caplan suggests, trust literatures, not individual research studies. By a “literature,” I mean the collective work conducted by many researchers, acting in decentralized fashion, to publish and circulate the results that will best persuade other researchers.

Second, treat research articles, or their popular media coverage, as possibilities to put in your mental toolbox rather than settled truths.

Literatures are more trustworthy than individual articles because they reflect a collective effort to establish reliable results. A supposed correlation gets refereed and scrutinized dozens of times, or maybe hundreds of times. If you have a new hypothesis, other researchers have a chance to make their names by knocking it down. There are also more eyes watching, in case real-world experience delivers results at odds with what a particular theory had been postulating. Or maybe there was a simple mistake in writing the computer code behind the paper’s result. Literatures contain a variety of different ways to come to a particular conclusion, and you can see whether they end up pointing in the same general direction.

You may not have time or the background to master a complete literature on a research topic, but these days you can send well-written prompts to GPT 5.2 Pro, Claude Opus 4.5, or Gemini 3.0 for some very good summaries of any literature you want. Furthermore, you can cross-check across these different AI models for additional reliability.

This is useful advice which is rarely heeded, and learning how to interpret a research literature is one of the most important skills in intellectual life.

Comments

Comments for this post are closed