I’ve been busy this past week working as part of a review committee. All the people involved in this are part of an organization that shares funding, and so participating is somewhat a requirement. The papers we choose aren’t going to count as refereed publications.This has turned out to mean that the overall quality of the submitted papers and reviews is quite a bit lower than you would expect at a top quality conference. I find this really quite sad and disheartening. It’s surprising how little effort people are willing to put into their work if they don’t think they’re going to get some great reward from it. It also probably doesn’t help that all the paper authors must be students, used loosely as it includes undergrads to post-docs and the reviewers are selected from the same group.
My role in this is slightly more important than your average reviewer. Instead of just reading and submitting a few reviews, I’m responsible for a set of papers. I had to select people to review them, remind them to actually review, read all the papers and reviews, and then create meta-reviews. In doing so, I had to determine which reviews should count more and which should be ignored. I needed to identify the key parts of the reviews that were salient and help distill all this information for the actual chairs of the committee to use to make their decision.
I didn’t think it was going to be as hard as it has turned out to be. I figured, with the number of people we asked to review (more than your average 3, since we had more than just the authors to get reviews from) that there would turn out to be a pretty good consensus on each paper, and that the reviewers would attempt to do a good job, and that they actually would take the time to understand the constraints on the papers that were submitted. And some of the papers are like that.
However, I also am seeing a really high number of “neutral” rankings of papers, which is really not useful for the committee. You might as well say I just don’t care. And it’s interesting to read the reviews associated with these. Some of them aren’t neutral at all, and so you wonder how they determined their score. Did they feel bad going lower?
In the group of papers I have, I have had exactly 1 review give a ranking below neutral. One! This is shocking, because I have one paper that I thought was really not well done, and yet it didn’t even get this ranking. Ugh. 😦
Anyway, the good (?) thing about going through all this is that it’s reminding me of a bunch of reviewing/writing pet peeves I have. So as I have been making my way through my stack of reading, I’m taking notes. Next week I’ll have a couple of posts on my pet peeves when it comes to papers and reviews.
Is there anything that drives you nuts when you read a paper or receive a review?