Tuesday, May 3, 2011

Peer review and transparency

There is some evidence that the status of your name or institution affects the conclusions of peer review:

There have been many studies of bias - with conflicting results - but the most famous was published in Behavioural and Brain Sciences [14]. The authors took 12 studies that came from prestigious institutions that had already been published in psychology journals. They retyped the papers, made minor changes to the titles, abstracts, and introductions but changed the authors’ names and institutions. They invented institutions with names like the Tri-Valley Center for Human Potential. The papers were then resubmitted to the journals that had first published them. In only three cases did the journals realise that they had already published the paper, and eight of the remaining nine were rejected - not because of lack of originality but because of poor quality. The authors concluded that this was evidence of bias against authors from less prestigious institutions.

The solution sometimes proposed is double-blind peer-review – where the referee does not know whose article he/she is reviewing – which is seen as a way of ensuring that famous names and well-known colleagues do not have an easier time getting published than others.  Daniel Lemire discusses a paper that found double-blind peer-review to actually hurts “outsiders” more than it leveled the playing field. Criticism also became harsher, and the quality increase was marginal at best.

Lemire concludes that transparency is better – interestingly, he makes the transparency argument against both the blinds in the double-blind: He seems to argue both that the author should be known to the referee, and that the referee and the review report should be known to the author:

But the best way to limit the biases is transparency, not more secrecy. Let the world know who rejected which paper and for what reasons.