Wednesday, May 4, 2011

An escape from uncertainty? On the support for peer review and hierarchical journals

Some time ago, after discussing peer-review and mailing the first quote from yesterday’s post to a colleague, he responded “OK, so design a better system, then.”
The challenge has been bouncing around in the back of my head for a while, when one day it hit me that (maybe) it is an impossible task – because the perceived benefits of the current system are illusory, while an important benefit of an alternative system would be that it was more transparent and thus would not provide the illusion of authoritativeness, finality and certainty.
Imagine a place where all articles could be published – an online repository of some sort. There’s A LOT of researchers out there, and there would be a flood of papers in any (even narrowly defined) field. You might see which ones other readers have read, you might even have tools in the repository for giving “starred reviews” (as on Amazon) but with scholarly comments, for giving evaluations of reviewers, and thus maybe even average “ratings weighted by how “useful/valid/perceptive” the reviewers have been judged,” and so on. There could be long comment and discussion threads, the different articles could be cross-referenced by researchers and readers, the whole thing could be in a “facebook-ish” system that made it harder to be an anonymous troll.
Even so – I think this would prove unsatisfactory to many (most?) researchers: I think there’s a human desire for someone to have the final say and state that “Yes – this is good, important and probably true!” It is a desire to have some external authority that can make the final judgment call that “your work is good!” or that “This result can be cited with confidence!” An open, transparent system makes it hard not to see the apes behind the machine. The “institution” of peer-reviewed, prestigious journals, in comparison, has a somewhat magical aura of authoritativeness and gravitas.
Put differently – the present journal system makes it easy to identify which “giants” we should stand on the shoulders of to see further, and it offers the hope that we can be published in a high-ranking journal and thus be future giants ourselves. A truly open system shows us that we are trying to build on the shoulders of  a large, shifting mass of more or less confused fellow ants all scrambling around trying to do the same thing.
I’m not sure how I can test this hunch – but if it’s correct then it will be difficult to move towards a more open access approach to science based primarily on post-review. Maybe it will change as new generations become more and more comfortable with on-line tools and evaluation methods, I don’t know. But my guess would be that you can marshal all the evidence you want against peer-review and tiered journals and it wouldn’t help. You could show that peer-review fails to catch errors, that referees are biased in favor of conclusions they like, that referees agree as often as two tossed coins, that it is a newfangled thing that was quite unusual in even top journals until the second half of the 20th century (think about it – they didn’t even have photocopiers in the “old days”), that a system of tiered journals creates publication bias in favor of spurious results, provides disincentives to replication studies, and so on and so forth.
Yes – a “top journal” may be just some guy acting as editor who gives two or three researchers access to an enormously impactful “Like-button,” but it doesn’t feel that way.

No comments:

Post a Comment