I’ve long been meaning to remark on my experiences of being on the SODA and ALENEX program committees this past year and fellow bloggers recent reflections on PC membership encourages me to finally post what little I have to say.

I went into the SODA PC expecting a lot of work.  I (mostly) cleared out a full month to do the reviews, which was not much less than the time allotted to finish the reviews.  Of the 48 papers [1] I had to review, I had approximately half sub-reviewed.  My sub-reviewers were amazing (timely, thorough, etc).  What I didn’t expect was the amount of time that went into discussing the papers.  With the virtual committee-meeting spanning roughly 3 weeks, I spent a lot of time going over those subreviewed papers, reading other PC member comments and adjusting or defending my opinion.  Since I didn’t plan for this, I didn’t have a lot of time to spend on it.  There was great potential to learn from the papers submitted and committee decisions, but the overwhelming amount of work didn’t leave time to synthesize conclusions at a higher level.  And then classes started up.  While it was a worthy experience, it doesn’t quite rise to the bar Claire experienced with her first SODA committee.

I think a physical PC meeting would have added the benefits that Bill highlights by developing a relationship with those members I didn’t previously know (which is many of them).  I still had the benefit of getting to know Dana Randall, who, although I have nothing to compare it to, did an amazing job and provided wonderful guidance along the way.

The ALENEX PC was much lower key.  13 papers to review.  And though getting subreviewers proved mostly futile (I think I managed to get 3 subreviewers – most of my emails for ALENEX were left unresponded to or turned down), the papers were slightly easier to review and I came out of the experience with a much better idea of what an experimental paper should look like.

I know our community has discussed the possibility of a tiered PC for STOFOSODA.  While I don’t know what the implications really would be, I can say that a less daunting experience (such as in a lower tier) would be preferable for a first-time PC experience.  I suppose one could emulate this by starting with workshop and smaller conference committees, but I think it is great that STOFOSODA has a history of including junior researchers on their PCs and one should hardly pick-and-choose when they are offered the chance to serve on their favourite conference’s committee.

[1] Yes, 48, oh ye in other fields with hierarchical committees.

I’m on the SODA 2011 program committee and finding it, as everything this last year (grant writing, NSF panelling, grad student advising), a condensed learning experience.

Our reviews are almost due which means that this newbie has been through all but a a few of my stack. I asked for subreviews for roughly half my stack and [drum roll] incredibly, despite the procrastinating reputation of computer scientists, my wonderful subreviewers returned all their reviews in to me on time (and several were early).¹

The most valuable aspect of these subreviews was not time-saving or expert-opinion-injecting (although that was greatly appreciated), but the stack of example reviews. Sure I’ve seen my share of reviews. But they were all reviews of my papers which has a certain bias. And sure, I’ve seen several reviews of other papers, but they were all reviews written by me. So, after reading this stack of reviews, I can only conclude that I have been a crappy conference subreviewer.

The review should obviously impart an opinion on the paper. But the ideal conference subreview should give enough details of the result to save the program committee member from having to read the paper (or at least, the entire paper). I’m pretty sure I’ve missed this idea writing subreviews in the past; I could cop this up to having never been taught how to write a review but I think it is more a failure of my ability to empathize with the program committee. I imagine I modelled my reviews on reviews of my own papers – in cases where such details were provided, I probably thought “This is boring. I know the results of my own paper, why is the reviewer telling me this?” – when they weren’t, I thought “why was my paper rejected?”². Embarrassing that it took so long for me to understand this process: subreviews are targeted (mostly) at the program committee member and (direct) reviews are terse because of the large paper load and the ability of a committee member to make an opinion without taking the time to write out a thorough review.

I think a nice addition to conference reviews would be the equivalent of the grant panel summary. Perhaps not so important for accepted papers, but for rejected papers it would be useful for authors to receive high-level constructive criticism (submit to conference X, fix these issues and it should get into conference Y, etc.) Of course, I don’t want to increase the work load of a program committee. At least not this year.

¹ One review was a day late but, after all, I could not process all the subreviews in one day, so I would hardly notice or mind.
² Funny, I don’t have a problem when a paper is accepted!