Reviewing a paper multiple times

It has happened several times to me now.

I (sub)review a paper for a conference and for one reason or another the paper is rejected.  The next conference deadline rolls around and I get a request to review the same paper.  I have never turned down these requests until today, and even today it was a “soft” turn-down.  I’ve experienced three characteristic situations:

  1. The first version has a bug.  I recommended a reject.  In every case that I can remember, the second version has the same bug, so I happily send the same review to the PC member, always being honest about reviewing this for a previous conference.
  2. I loved the first version and recommended an accept.  I happily send the same review, perhaps with a little extra “sugar” in the hopes that it will be accepted this time, always being honest about reviewing this for a previous conference.
  3. I was luke warm about the first version.  I check to see that the paper hasn’t changed.  It hasn’t.  Should I submit the same luke-warm review?  I did this time, but was pretty strong about not really thinking it would be useful to the committee.

I’m starting to think that a second review by the same person for a different committee isn’t very useful.  Except perhaps in the first case when a bug is a bug is a bug.  (I had one situation where I reviewed an un-changing, incorrect paper three times only to see it one day be accepted to a conference – with the bug included – for which I was not a reviewer.)

So, my question is: what do you do in these situations?  As a PC member, do you welcome “re”reviews?  In all situations?  Or just some?

Print Friendly, PDF & Email

6 thoughts on “Reviewing a paper multiple times

  1. D. Eppstein

    Re-reviewing has the disadvantage that a single influential reviewer can blackball a paper from many conferences based on an irrational dislike to the paper. On the other hand, if the dislike is not irrational, this may not be a disadvantage…

    Re-reviewing has the advantage that it encourages authors to take reviews seriously and actually revise their papers rather than just resubmitting and hoping for less careful reviewers next time.

    So I don’t know, but I usually err on the side of re-reviewing.

  2. Alex Lopez-Ortiz

    How about:

    4. The weak-reject SODA/STOC/FOCS paper looks the same, but now it is submitted to a lower quality venue.

    A re-reviewer can quickly upgrade the score to weak accept and save reviewer cycles.

    In this case a re-review is rather useful.

  3. Dave P

    I also had a buglike experience like yours, I saw it twice and I guess the third time was a charm, for which I wasn’t involved. On the other hand, for another paper once I submitted a subreview and the middleman noticed I made a faulty claim of a bug.

    Back on the first hand, it bugged me a lot that they didn’t even incorporate trivial typo/grammar corrections, and in that case that adds to my confidence that they paid not so much attention.

    I always wonder if there should be a secret illuminati-style mega-database of old reviews, to save PCs from the hassle of extra work. Seeing what a previous PC said/did seems to be valuable… a good situation I could imagine is the paper gets 1 or 2 fresh reviews for each conference and 1 or 2 old ones. Bonus points if it could also provide a diff tool that could parse .pdfs and latex formulas.

  4. steve uurtamo

    Just to turn this around — two equivalently “good” conferences may give vastly different results for a nearly unchanged paper. It’s crazy to expect different results every time you offer the same information, but this happens because there is a high variance in review quality, even at top conferences.

  5. Carlos Linares López

    Very interesting question! (found it in twitter and could not refrain from posting here my experience …)

    I was assigned once a paper about a very interesting idea! However, it *seemed to me* that the solution was rather poor so I rejected the paper and gave extensive arguments. However, the other two reviewers actually recommended acceptance. Fortunately, one of the other reviewers engaged in a large discussion; other PC members got in as well and everyone recommended acceptance. I finally gave up and recommended rejection under the observation that so many people was in favour … But the Senior PC rejected the paper under the assumption that one rejection (like mine) was enough to reject the paper.

    I felt so sad about that that when I saw the paper again submitted to another top conference in my field (artificial intelligence) I marked a conflict of interest in the bidding phase! After the reviewing process and as soon as the chairs published the accepted papers I checked the list to see that the paper was not there. Sincerely, I felt some relief (hope you understand …)

    Finally, I saw the paper accepted in a minor conference.

    In all cases the solution posted was always the same (I assume it was the same in the second conference since I saw the same schema in the third and minor conference). In this particular conference, I *think* (but would be happy to hear about your thoughts) I did the right thing: I felt like being an obstacle and in spite of the quality of any work, there are always people behind. Do not want to harm them!


  6. Mohammad

    As a subreferee, I always resend my old review (checking for changes in the paper, and adjusting for conference quality) and as a PC member, I accept such reviews. I think the disadvantage of re-using reviews is outweighed by the saving in reviewing costs.

Comments are closed.