[With] open-source reviewing…the journal posts a submitted paper online and allows not just assigned reviewers but anyone to critique it. After a few weeks, the author revises, the editors accept or reject and the journal posts all, including the editors’ rationale…
Open, collaborative review may seem a scary departure. But scientists might find it salutary. It stands to maintain rigor, turn review processes into productive forums and make publication less a proprietary claim to knowledge than the spark of a fruitful exchange. And if collaborative review can’t prevent fraud, it seems certain to discourage it, since shady scientists would have to tell their stretchers in public. Hwang’s fabrications, as it happens, were first uncovered in Web exchanges among scientists who found his data suspicious. Might that have happened faster if such examination were built into the publishing process? "Never underestimate competitors," Delamothe says, for they are motivated. Science – and science – might have dodged quite a headache by opening Hwang’s work to wider prepublication scrutiny.
Here is a bit more. What might be some arguments against this practice?
1. It is too easily manipulated by your friends, or perhaps by your enemies.
2. The resulting morass of comments must be interpreted. We are back to editorial discretion, but it is better to have some referees rather than none.
3. The purpose of journals is not to always make the right decision, but rather to certify the quality of outstanding work to more general audiences. By blurring the evaluation process, open source reviewing would make journals as a whole less reliable.
4. Don’t we already have this option? I could post a paper on this blog, open up the comments, and receive a call from the AER, asking for a submission. I guess my answering machine isn’t working.
5. The current system allows for editorial manipulation through the choice of referees. This is good. An innovator needs only to convince a single editor, not a jackal-like pack of seething commentators [hey guys, that’s you!].
What is the goal of publishing anyway? To assign "just outcomes"? To make sure that the one percent of worthwhile papers find a prestigious outlet? To provide incentives for those papers to be written in the first place? To increase the prestige of science as a whole? Since I don’t understand why on-line publishing hasn’t already taken over, this scheme is hard to evaluate. Comments are open….