It has happened several times to me now.
I (sub)review a paper for a conference and for one reason or another the paper is rejected. The next conference deadline rolls around and I get a request to review the same paper. I have never turned down these requests until today, and even today it was a “soft” turn-down. I’ve experienced three characteristic situations:
- The first version has a bug. I recommended a reject. In every case that I can remember, the second version has the same bug, so I happily send the same review to the PC member, always being honest about reviewing this for a previous conference.
- I loved the first version and recommended an accept. I happily send the same review, perhaps with a little extra “sugar” in the hopes that it will be accepted this time, always being honest about reviewing this for a previous conference.
- I was luke warm about the first version. I check to see that the paper hasn’t changed. It hasn’t. Should I submit the same luke-warm review? I did this time, but was pretty strong about not really thinking it would be useful to the committee.
I’m starting to think that a second review by the same person for a different committee isn’t very useful. Except perhaps in the first case when a bug is a bug is a bug. (I had one situation where I reviewed an un-changing, incorrect paper three times only to see it one day be accepted to a conference – with the bug included – for which I was not a reviewer.)
So, my question is: what do you do in these situations? As a PC member, do you welcome “re”reviews? In all situations? Or just some?
Re-reviewing has the disadvantage that a single influential reviewer can blackball a paper from many conferences based on an irrational dislike to the paper. On the other hand, if the dislike is not irrational, this may not be a disadvantage…
Re-reviewing has the advantage that it encourages authors to take reviews seriously and actually revise their papers rather than just resubmitting and hoping for less careful reviewers next time.
So I don’t know, but I usually err on the side of re-reviewing.
How about:
4. The weak-reject SODA/STOC/FOCS paper looks the same, but now it is submitted to a lower quality venue.
A re-reviewer can quickly upgrade the score to weak accept and save reviewer cycles.
In this case a re-review is rather useful.
I also had a buglike experience like yours, I saw it twice and I guess the third time was a charm, for which I wasn’t involved. On the other hand, for another paper once I submitted a subreview and the middleman noticed I made a faulty claim of a bug.
Back on the first hand, it bugged me a lot that they didn’t even incorporate trivial typo/grammar corrections, and in that case that adds to my confidence that they paid not so much attention.
I always wonder if there should be a secret illuminati-style mega-database of old reviews, to save PCs from the hassle of extra work. Seeing what a previous PC said/did seems to be valuable… a good situation I could imagine is the paper gets 1 or 2 fresh reviews for each conference and 1 or 2 old ones. Bonus points if it could also provide a diff tool that could parse .pdfs and latex formulas.
Just to turn this around — two equivalently “good” conferences may give vastly different results for a nearly unchanged paper. It’s crazy to expect different results every time you offer the same information, but this happens because there is a high variance in review quality, even at top conferences.
Very interesting question! (found it in twitter and could not refrain from posting here my experience …)
I was assigned once a paper about a very interesting idea! However, it *seemed to me* that the solution was rather poor so I rejected the paper and gave extensive arguments. However, the other two reviewers actually recommended acceptance. Fortunately, one of the other reviewers engaged in a large discussion; other PC members got in as well and everyone recommended acceptance. I finally gave up and recommended rejection under the observation that so many people was in favour … But the Senior PC rejected the paper under the assumption that one rejection (like mine) was enough to reject the paper.
I felt so sad about that that when I saw the paper again submitted to another top conference in my field (artificial intelligence) I marked a conflict of interest in the bidding phase! After the reviewing process and as soon as the chairs published the accepted papers I checked the list to see that the paper was not there. Sincerely, I felt some relief (hope you understand …)
Finally, I saw the paper accepted in a minor conference.
In all cases the solution posted was always the same (I assume it was the same in the second conference since I saw the same schema in the third and minor conference). In this particular conference, I *think* (but would be happy to hear about your thoughts) I did the right thing: I felt like being an obstacle and in spite of the quality of any work, there are always people behind. Do not want to harm them!
Cheers,
As a subreferee, I always resend my old review (checking for changes in the paper, and adjusting for conference quality) and as a PC member, I accept such reviews. I think the disadvantage of re-using reviews is outweighed by the saving in reviewing costs.