(Ahhh – I even stayed up late last night writing this post and then, apparently, didn’t hit schedule. Sorry.)
The longer I’m a grad student (although could probably just be worded as the longer I’m in academia) the better(?) I get at tearing papers apart. I think this is the real reason you’re not suppose to look back at earlier work. 😛 And there are papers that I read, that have been published, where I wonder how they convinced people that their results were really that important/significant. Especially when they usually have a total of less than 20 test subjects (including control and experimental). But they happen.
However, this link has been floating around recently. It’s a blog post by a professor, Rolf Zwaan, and it tears about apart a paper that has been recently published. He does so, someone nicely (in that his language isn’t excessively harsh) by asking 50 questions about the paper. Many are about methodology. Some are about details that weren’t provided and he’d like to know more. But it is a pretty methodical process of shredding a paper.
Now, I haven’t read the actual paper, so I’m not saying his claims are or are not valid (although they like are). It’s also not in my area, so I don’t know any of the related work – or even how papers are judged over in psychology. I do know, that at least some of the questions he raises I would’ve thought about (just based on both how I read papers now and information I look for based on how I write papers).
More, what I find interesting/fascinating/horrifying about the whole thing, is how public it is. I’m not the first person to link to his post. He admits that it’s been widely shared (in the comments). But this means, that those authors, who likely worked very hard to do the research, are been widely identified and shredded. I know, if it was me, I’d be horrified, and probably leave academia (but maybe I just have a really thin skin). And, since I can tear apart one of my papers (whoops on the statistics) I hope no one ever finds it and decides to do so to me.
I do wonder what I would do if I found a paper, that I thought was absolutely terrible, had been published. I’m not sure I’d do anything. And I’m not sure that there really is a “right” way to respond to this problem. Terrible papers exist. We all know that. And we’ve all read some. And I, personally, think too much is being published as is (which is part of the reason for the terrible papers – the bar seems to be continually sinking).
The only solution I can see, is to be the best reviewer you can possibly be. And this means reading papers carefully. And asking the tough questions. And pointing out if there was an area that you were not qualified to judge in hopes that someone else will be found to judge that section. But, this only works if you’re assigned as a reviewer of a paper. Once a paper is published, it’s too late.
Sure, papers can be retracted – but that seems to be a very rare thing (although I’m interested in doing some research on this topic now – so maybe I’ll post about it in more detail). You can write to the authors, but don’t expect that to change anything. You can tell your colleagues and friends, but that won’t change anything either. You can post about the paper on your very public blog, but it’s doubtful that will change anything besides more people piling on with problems.
Really, all around, I find the whole thing just so unfortunate. I feel for the authors. But I also don’t like (potentially) crappy work. I understand the blog authors frustrations. But I’m not one for airing all these problems quite so publicly. Sigh.
If it was you, what would you do? Either as an author who is being called out or as a reader who finds such a paper? Or, even better, what have you done before?