# Why Most Published Research Findings are False

Sadly, things get really bad when lots of researchers are chasing the

same set of hypotheses. Indeed, the larger the number of researchers

the more likely the average result is to be false! The easiest way to

see this is to note that when we have lots of researchers every true

hypothesis will be found to be true but eventually so will every false

hypothesis. Thus, as the number of researchers increases, the

probability that a given result is true goes to the probability in the

population, in my example 200/1000 or 20 percent.

A meta analysis will go some way to fixing the last problem so the

point is not that knowledge declines with the number of researchers but

rather that with lots of researchers every crackpot theory will have at

least one scientific study that it can cite in it's support.

The meta analysis approach, however, will work well only if the results that are published reflect the results that are discovered. But

editors and referees (and authors too) like results which reject the

null – i.e. they want to see a theory that is supported not a paper

that says we tried this and this and found nothing (which seems like an admission

of failure).

Brad DeLong and Kevin Lang wrote a classic paper suggesting

that one of the few times that journals will accept a paper that fails

to reject the null is when the *evidence against the null is strong *(and

thus failing to reject the null is considered surprising and

important). DeLong and Lang show that this can result in a paradox. Taken on its own, a paper which fails to reject the null provides evidence in favor of the null, i.e. against the alternative hypothesis and so should increase the probability that a rational person thinks the null is true. But when a rational person takes into account the selection effect, the fact that the only time papers which fail to reject the null are published is when the evidence against the null is strong, the *publication* of a paper failing to reject the null can cause him to increase his belief in the alternative theory!

What can be done about these problems? (Some cribbed straight from Ioannidis and some my own suggestions.)

1) In evaluating any study try to take into account the amount of background noise. That is, remember that the more hypotheses which are tested and the less selection which goes into choosing hypotheses the more likely it is that you are looking at noise.

2) Bigger samples are better. (But note that even big samples won't help to solve the problems of observational studies which is a whole other problem).

3) Small effects are to be distrusted.

4) Multiple sources and types of evidence are desirable.

5) Evaluate literatures not individual papers.

6) Trust empirical papers which test other people's theories more than empirical papers which test the author's theory.

7) As an editor or referee, don't reject papers that fail to reject the null.