John Bohannon, a visiting scholar at Harvard with a PhD in molecular biology recently performed a study to check how well these new “open-access online journals” were performing peer review. And the short answer, is that most of them aren’t. An article about his study can be found here.
But, a quick summary is that Bohannon wrote a paper with serious (and obvious) scientific flaws in it. He then sent the paper to 300 different scientific journals. Over half of the journals accepted the paper. One, did perform peer review that pointed out the flaws in the paper. But, Bohannon did his “final edits” (he actually didn’t change anything) and it was accepted as is. And of the 300 journals, only one actually identified (or at least openly flagged) the ethical and research flaws.
It does make you wonder how well our peer review system works. Now, these journals are not the big-named huge-impact-factor ones we’d all like to someday get a paper published in (think Nature). But they are gaining traction. I feel like I get calls to submit to these types of journals every day (not quite, but it definitely feels like it). And most of these calls for papers have absolutely nothing to do with my research area.
I actually have a paper that got accepted (after actual peer review) that we later discovered had flaws in the analysis. The overall outcomes of the experiment did not change with the corrected analysis, but it does mean that this paper is out there – flaws and all. And that three other members of the scientific community did not see the problem.
Of course, part of this problem, is that while we may all be experts (or becoming experts) in our particular area, few of us actually have enough background in statistics and analyzing experiments to actually notice some of these errors when it happens. And even when we do have the knowledge, there becomes the problem of standing up and saying something and having someone take you seriously (this obviously shouldn’t be a problem, but unfortunately, in my experience, it is). And there’s that weird blindness that seems to affect people when regarding specific other people’s work (you know – those scientists that some people feel can do no wrong). Other times, I think it’s that people become tired. Tired of correcting the same error over and over and over again. And tired of saying something only to be ignored.
Then, of course, there are also those scientists who want to have their own work peer reviewed but refuse to write more than a sentence or two when they review others. These are some of the people in the process that cause the largest problem. You can end up going from three actual reviews down to only one or two – which is not really enough information for making an educated evaluation of the paper.
What has your experience with peer review been like? Have you found it generally helpful? Have you experienced errors that you’ve found in your own work that reviewers didn’t catch?