Why Most Published Research Findings are False

Sadly, things get really bad when lots of researchers are chasing the
same set of hypotheses.  Indeed, the larger the number of researchers
the more likely the average result is to be false!  The easiest way to
see this is to note that when we have lots of researchers every true
hypothesis will be found to be true but eventually so will every false
hypothesis.  Thus, as the number of researchers increases, the
probability that a given result is true goes to the probability in the
population, in my example 200/1000 or 20 percent.

A meta analysis will go some way to fixing the last problem so the
point is not that knowledge declines with the number of researchers but
rather that with lots of researchers every crackpot theory will have at
least one scientific study that it can cite in it's support.

The meta analysis approach, however, will work well only if the results that are published reflect the results that are discovered.  But
editors and referees (and authors too) like results which reject the
null – i.e. they want to see a theory that is supported not a paper
that says we tried this and this and found nothing (which seems like an admission
of failure). 

Brad DeLong and Kevin Lang wrote a classic paper suggesting
that one of the few times that journals will accept a paper that fails
to reject the null is when the evidence against the null is strong (and
thus failing to reject the null is considered surprising and
important).  DeLong and Lang show that this can result in a paradox.  Taken on its own, a paper which fails to reject the null provides evidence in favor of the null, i.e. against the alternative hypothesis and so should increase the probability that a rational person thinks the null is true.  But when a rational person takes into account the selection effect, the fact that the only time papers which fail to reject the null are published is when the evidence against the null is strong, the publication of a paper failing to reject the null can cause him to increase his belief in the alternative theory!

What can be done about these problems?  (Some cribbed straight from Ioannidis and some my own suggestions.)

1)  In evaluating any study try to take into account the amount of background noise.  That is, remember that the more hypotheses which are tested and the less selection which goes into choosing hypotheses the more likely it is that you are looking at noise.

2) Bigger samples are better.  (But note that even big samples won't help to solve the problems of observational studies which is a whole other problem). 

3) Small effects are to be distrusted.

4) Multiple sources and types of evidence are desirable.

5) Evaluate literatures not individual papers.

6)  Trust empirical papers which test other people's theories more than empirical papers which test the author's theory.

7)  As an editor or referee, don't reject papers that fail to reject the null.