One paper. 50 questions.

(Ahhh – I even stayed up late last night writing this post and then, apparently, didn’t hit schedule. Sorry.)

The longer I’m a grad student (although could probably just be worded as the longer I’m in academia) the better(?) I get at tearing papers apart. I think this is the real reason you’re not suppose to look back at earlier work. 😛 And there are papers that I read, that have been published, where I wonder how they convinced people that their results were really that important/significant. Especially when they usually have a total of less than 20 test subjects (including control and experimental). But they happen.

However, this link has been floating around recently. It’s a blog post by a professor, Rolf Zwaan, and it tears about apart a paper that has been recently published. He does so, someone nicely (in that his language isn’t excessively harsh) by asking 50 questions about the paper. Many are about methodology. Some are about details that weren’t provided and he’d like to know more. But it is a pretty methodical process of shredding a paper.

Now, I haven’t read the actual paper, so I’m not saying his claims are or are not valid (although they like are). It’s also not in my area, so I don’t know any of the related work – or even how papers are judged over in psychology. I do know, that at least some of the questions he raises I would’ve thought about (just based on both how I read papers now and information I look for based on how I write papers).

More, what I find interesting/fascinating/horrifying about the whole thing, is how public it is. I’m not the first person to link to his post. He admits that it’s been widely shared (in the comments). But this means, that those authors, who likely worked very hard to do the research, are been widely identified and shredded. I know, if it was me, I’d be horrified, and probably leave academia (but maybe I just have a really thin skin). And, since I can tear apart one of my papers (whoops on the statistics) I hope no one ever finds it and decides to do so to me.

I do wonder what I would do if I found a paper, that I thought was absolutely terrible, had been published. I’m not sure I’d do anything. And I’m not sure that there really is a “right” way to respond to this problem. Terrible papers exist. We all know that. And we’ve all read some. And I, personally, think too much is being published as is (which is part of the reason for the terrible papers – the bar seems to be continually sinking).

The only solution I can see, is to be the best reviewer you can possibly be. And this means reading papers carefully. And asking the tough questions. And pointing out if there was an area that you were not qualified to judge in hopes that someone else will be found to judge that section. But, this only works if you’re assigned as a reviewer of a paper. Once a paper is published, it’s too late.

Sure, papers can be retracted – but that seems to be a very rare thing (although I’m interested in doing some research on this topic now – so maybe I’ll post about it in more detail). You can write to the authors, but don’t expect that to change anything. You can tell your colleagues and friends, but that won’t change anything either. You can post about the paper on your very public blog, but it’s doubtful that will change anything besides more people piling on with problems.

Really, all around, I find the whole thing just so unfortunate. I feel for the authors. But I also don’t like (potentially) crappy work. I understand the blog authors frustrations. But I’m not one for airing all these problems quite so publicly. Sigh.

If it was you, what would you do? Either as an author who is being called out or as a reader who finds such a paper? Or, even better, what have you done before?


2 thoughts on “One paper. 50 questions.

  1. One of the things that interests me in talking to more experienced scientists is that in the end papers that you review may get published without all of the changes you recommended. Sometimes and editor may give the paper a pass and publish it anyway. In other cases, you may make recommendations but not see the result of those suggestions before publication.

    I’ve heard this scenario as a reason to stay anonymous as a reviewer — just in case the published result isn’t something you have really approved of in the end. I have yet to review a paper, but it’s something I’ll have to put some thought into when deciding whether to reveal my name or not.

    • I’m not sure how I feel about attaching names to reviews. I can see potential benefits (maybe if you knew you were being named, people would put more effort into their reviews). But, there’s the downside, like you pointed out – that perhaps it gets published over something you disagree strongly with. Or even, that the authors takes a lot of offence and then possibly takes it out on you in the future.

      For now, I still thing the double blind review system is the best we have. Even though, at the end of the day – the people running the conference/journal know exactly who everyone is.

Join the discussion

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s